2024-02-09T21:30:50.911 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-09T21:30:50.911 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi/7554325 --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --name yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi --verbose --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi/7554325/orig.config.yaml 2024-02-09T21:30:50.964 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-09T21:30:51.053 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi/7554325 branch: main description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7554325' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 3558 sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 sleep_before_teardown: 0 subset: 111/120000 suite: fs suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 targets: smithi006.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNL5Lq/5A0kvQ8OuALLL7kAYuBa8yO7d4VjvaOmaK7KrVmb52FFr2ssHWu7VhEHtmrdm/qgHO3FAWCgEk8BaI5I= smithi090.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJwJ+E73YPLZm1OxWewpM2ZDiu1pmnGL7rta489O02oIAeHn7wmSYJk3TjQfoEv0GAtghOyLMLo5s92pNNwiaJo= tasks: - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-09_19:35:09 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-02-09T21:30:51.053 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa; will attempt to use it 2024-02-09T21:30:51.054 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks 2024-02-09T21:30:51.054 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-09T21:30:51.055 INFO:teuthology.task.internal:Checking packages... 2024-02-09T21:30:51.079 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90' 2024-02-09T21:30:51.080 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-09T21:30:51.080 INFO:teuthology.packaging:ref: None 2024-02-09T21:30:51.080 INFO:teuthology.packaging:tag: None 2024-02-09T21:30:51.080 INFO:teuthology.packaging:branch: main 2024-02-09T21:30:51.080 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T21:30:51.080 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-02-09T21:30:51.369 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1361.g290212a1 2024-02-09T21:30:51.371 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-09T21:30:51.378 INFO:teuthology.task.internal:no buildpackages task found 2024-02-09T21:30:51.378 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-09T21:30:51.442 INFO:teuthology.task.internal:Saving configuration 2024-02-09T21:30:51.465 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-09T21:30:51.492 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-09T21:30:51.516 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi/7554325', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-09 21:25:11.194102', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNL5Lq/5A0kvQ8OuALLL7kAYuBa8yO7d4VjvaOmaK7KrVmb52FFr2ssHWu7VhEHtmrdm/qgHO3FAWCgEk8BaI5I='} 2024-02-09T21:30:51.540 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi/7554325', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-09 21:25:11.196105', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJwJ+E73YPLZm1OxWewpM2ZDiu1pmnGL7rta489O02oIAeHn7wmSYJk3TjQfoEv0GAtghOyLMLo5s92pNNwiaJo='} 2024-02-09T21:30:51.540 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-09T21:30:51.547 INFO:teuthology.task.internal:roles: ubuntu@smithi006.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-02-09T21:30:51.547 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-02-09T21:30:51.547 INFO:teuthology.run_tasks:Running task console_log... 2024-02-09T21:30:51.657 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe26f096790>, signals=[15]) 2024-02-09T21:30:51.658 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-09T21:30:51.664 INFO:teuthology.task.internal:Opening connections... 2024-02-09T21:30:51.664 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi006.front.sepia.ceph.com 2024-02-09T21:30:51.666 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T21:30:51.817 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi090.front.sepia.ceph.com 2024-02-09T21:30:51.818 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T21:30:51.908 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-09T21:30:51.915 DEBUG:teuthology.orchestra.run.smithi006:> uname -m 2024-02-09T21:30:51.933 INFO:teuthology.orchestra.run.smithi006.stdout:x86_64 2024-02-09T21:30:51.934 DEBUG:teuthology.orchestra.run.smithi006:> cat /etc/os-release 2024-02-09T21:30:51.992 INFO:teuthology.orchestra.run.smithi006.stdout:NAME="CentOS Stream" 2024-02-09T21:30:51.992 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION="9" 2024-02-09T21:30:51.992 INFO:teuthology.orchestra.run.smithi006.stdout:ID="centos" 2024-02-09T21:30:51.992 INFO:teuthology.orchestra.run.smithi006.stdout:ID_LIKE="rhel fedora" 2024-02-09T21:30:51.992 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_ID="9" 2024-02-09T21:30:51.993 INFO:teuthology.orchestra.run.smithi006.stdout:PLATFORM_ID="platform:el9" 2024-02-09T21:30:51.993 INFO:teuthology.orchestra.run.smithi006.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-09T21:30:51.993 INFO:teuthology.orchestra.run.smithi006.stdout:ANSI_COLOR="0;31" 2024-02-09T21:30:51.993 INFO:teuthology.orchestra.run.smithi006.stdout:LOGO="fedora-logo-icon" 2024-02-09T21:30:51.993 INFO:teuthology.orchestra.run.smithi006.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-09T21:30:51.993 INFO:teuthology.orchestra.run.smithi006.stdout:HOME_URL="https://centos.org/" 2024-02-09T21:30:51.993 INFO:teuthology.orchestra.run.smithi006.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-09T21:30:51.993 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-09T21:30:51.993 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-09T21:30:51.994 INFO:teuthology.lock.ops:Updating smithi006.front.sepia.ceph.com on lock server 2024-02-09T21:30:52.014 DEBUG:teuthology.orchestra.run.smithi090:> uname -m 2024-02-09T21:30:52.032 INFO:teuthology.orchestra.run.smithi090.stdout:x86_64 2024-02-09T21:30:52.032 DEBUG:teuthology.orchestra.run.smithi090:> cat /etc/os-release 2024-02-09T21:30:52.089 INFO:teuthology.orchestra.run.smithi090.stdout:NAME="CentOS Stream" 2024-02-09T21:30:52.089 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION="9" 2024-02-09T21:30:52.089 INFO:teuthology.orchestra.run.smithi090.stdout:ID="centos" 2024-02-09T21:30:52.089 INFO:teuthology.orchestra.run.smithi090.stdout:ID_LIKE="rhel fedora" 2024-02-09T21:30:52.089 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_ID="9" 2024-02-09T21:30:52.089 INFO:teuthology.orchestra.run.smithi090.stdout:PLATFORM_ID="platform:el9" 2024-02-09T21:30:52.089 INFO:teuthology.orchestra.run.smithi090.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-09T21:30:52.089 INFO:teuthology.orchestra.run.smithi090.stdout:ANSI_COLOR="0;31" 2024-02-09T21:30:52.089 INFO:teuthology.orchestra.run.smithi090.stdout:LOGO="fedora-logo-icon" 2024-02-09T21:30:52.089 INFO:teuthology.orchestra.run.smithi090.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-09T21:30:52.089 INFO:teuthology.orchestra.run.smithi090.stdout:HOME_URL="https://centos.org/" 2024-02-09T21:30:52.090 INFO:teuthology.orchestra.run.smithi090.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-09T21:30:52.090 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-09T21:30:52.090 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-09T21:30:52.090 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server 2024-02-09T21:30:52.114 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-09T21:30:52.123 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-09T21:30:52.160 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-09T21:30:52.160 DEBUG:teuthology.orchestra.run.smithi006:> test '!' -e /home/ubuntu/cephtest 2024-02-09T21:30:52.163 DEBUG:teuthology.orchestra.run.smithi090:> test '!' -e /home/ubuntu/cephtest 2024-02-09T21:30:52.178 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-09T21:30:52.184 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-09T21:30:52.184 DEBUG:teuthology.orchestra.run.smithi006:> test -z $(ls -A /var/lib/ceph) 2024-02-09T21:30:52.220 DEBUG:teuthology.orchestra.run.smithi090:> test -z $(ls -A /var/lib/ceph) 2024-02-09T21:30:52.255 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-09T21:30:52.310 INFO:teuthology.run_tasks:Running task kernel... 2024-02-09T21:30:52.325 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-09T21:30:52.325 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-09T21:30:52.326 DEBUG:teuthology.orchestra.run.smithi006:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-09T21:30:52.326 DEBUG:teuthology.orchestra.run.smithi090:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-09T21:30:52.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T21:30:52.343 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2024-02-09T21:30:52.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T21:30:52.344 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-02-09T21:30:52.401 INFO:teuthology.orchestra.run.smithi090.stdout:5.14.0-361.el9.x86_64 2024-02-09T21:30:52.401 INFO:teuthology.task.kernel:Running kernel on smithi090: 5.14.0-361.el9.x86_64 2024-02-09T21:30:52.401 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum install -y kernel 2024-02-09T21:30:52.402 INFO:teuthology.orchestra.run.smithi006.stdout:5.14.0-361.el9.x86_64 2024-02-09T21:30:52.402 INFO:teuthology.task.kernel:Running kernel on smithi006: 5.14.0-361.el9.x86_64 2024-02-09T21:30:52.402 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum install -y kernel 2024-02-09T21:30:54.610 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-02-09T21:30:54.987 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-02-09T21:30:56.177 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - BaseOS 5.1 MB/s | 2.0 MB 00:00 2024-02-09T21:30:56.413 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.0 MB 00:00 2024-02-09T21:30:57.001 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-02-09T21:30:57.290 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-02-09T21:30:59.347 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-02-09T21:31:05.098 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 5.9 MB 00:02 2024-02-09T21:31:06.774 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 15 kB 00:00 2024-02-09T21:31:07.703 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 26 MB/s | 20 MB 00:00 2024-02-09T21:31:12.511 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-02-09T21:31:14.802 INFO:teuthology.orchestra.run.smithi090.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-09T21:31:14.802 INFO:teuthology.orchestra.run.smithi090.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-09T21:31:14.917 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-09T21:31:14.922 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:31:14.922 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-02-09T21:31:14.922 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:31:14.922 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-09T21:31:14.922 INFO:teuthology.orchestra.run.smithi090.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-09T21:31:14.922 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-02-09T21:31:14.923 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-09T21:31:14.923 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-09T21:31:14.923 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-09T21:31:14.923 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:31:14.923 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-09T21:31:14.923 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:31:14.923 INFO:teuthology.orchestra.run.smithi090.stdout:Install 4 Packages 2024-02-09T21:31:14.923 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:31:14.924 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 99 M 2024-02-09T21:31:14.924 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 122 M 2024-02-09T21:31:14.924 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-09T21:31:15.280 INFO:teuthology.orchestra.run.smithi090.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 17 MB/s | 6.0 MB 00:00 2024-02-09T21:31:15.855 INFO:teuthology.orchestra.run.smithi090.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 22 MB/s | 21 MB 00:00 2024-02-09T21:31:16.647 INFO:teuthology.orchestra.run.smithi090.stdout:(3/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 23 MB/s | 39 MB 00:01 2024-02-09T21:31:16.997 INFO:teuthology.orchestra.run.smithi090.stdout:(4/4): kernel-modules-core-5.14.0-419.el9.x86_6 19 MB/s | 33 MB 00:01 2024-02-09T21:31:17.001 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:31:17.001 INFO:teuthology.orchestra.run.smithi090.stdout:Total 48 MB/s | 99 MB 00:02 2024-02-09T21:31:17.001 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-09T21:31:17.397 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-09T21:31:17.398 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-09T21:31:18.378 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-09T21:31:18.378 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-09T21:31:20.843 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-09T21:31:21.547 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-09T21:31:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-09T21:31:24.460 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-09T21:31:24.656 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-09T21:31:29.369 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-09T21:31:29.479 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-09T21:31:34.223 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-09T21:32:04.742 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-09T21:32:04.742 INFO:teuthology.orchestra.run.smithi090.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-09T21:32:04.742 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:32:04.813 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-09T21:32:05.354 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-09T21:32:05.355 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-09T21:32:05.355 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-09T21:32:05.355 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-09T21:32:05.841 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-09T21:32:05.842 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:32:05.842 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-09T21:32:05.842 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-09T21:32:05.842 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-09T21:32:05.842 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-09T21:32:05.842 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-09T21:32:05.842 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:32:05.842 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-09T21:32:06.400 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q kernel | sort -rV | head -n 1 2024-02-09T21:32:06.542 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-09T21:32:06.542 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-09T21:32:06.542 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-09T21:32:06.543 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-09T21:32:06.543 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-09T21:32:06.543 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-02-09T21:32:06.543 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi090.front.sepia.ceph.com, path=None, version=distro) 2024-02-09T21:32:06.543 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q kernel | sort -rV | head -n 1 2024-02-09T21:32:06.650 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-09T21:32:06.650 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-09T21:32:06.650 DEBUG:teuthology.orchestra.run.smithi090:> sudo rpm -qi grub2-tools 2024-02-09T21:32:06.684 INFO:teuthology.orchestra.run.smithi090.stdout:Name : grub2-tools 2024-02-09T21:32:06.684 INFO:teuthology.orchestra.run.smithi090.stdout:Epoch : 1 2024-02-09T21:32:06.684 INFO:teuthology.orchestra.run.smithi090.stdout:Version : 2.06 2024-02-09T21:32:06.684 INFO:teuthology.orchestra.run.smithi090.stdout:Release : 68.el9 2024-02-09T21:32:06.684 INFO:teuthology.orchestra.run.smithi090.stdout:Architecture: x86_64 2024-02-09T21:32:06.684 INFO:teuthology.orchestra.run.smithi090.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-09T21:32:06.685 INFO:teuthology.orchestra.run.smithi090.stdout:Group : Unspecified 2024-02-09T21:32:06.685 INFO:teuthology.orchestra.run.smithi090.stdout:Size : 8264254 2024-02-09T21:32:06.685 INFO:teuthology.orchestra.run.smithi090.stdout:License : GPLv3+ 2024-02-09T21:32:06.685 INFO:teuthology.orchestra.run.smithi090.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-09T21:32:06.685 INFO:teuthology.orchestra.run.smithi090.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-09T21:32:06.685 INFO:teuthology.orchestra.run.smithi090.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-09T21:32:06.685 INFO:teuthology.orchestra.run.smithi090.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-09T21:32:06.685 INFO:teuthology.orchestra.run.smithi090.stdout:Packager : builder@centos.org 2024-02-09T21:32:06.685 INFO:teuthology.orchestra.run.smithi090.stdout:Vendor : CentOS 2024-02-09T21:32:06.685 INFO:teuthology.orchestra.run.smithi090.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-09T21:32:06.685 INFO:teuthology.orchestra.run.smithi090.stdout:Summary : Support tools for GRUB. 2024-02-09T21:32:06.685 INFO:teuthology.orchestra.run.smithi090.stdout:Description : 2024-02-09T21:32:06.685 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:32:06.685 INFO:teuthology.orchestra.run.smithi090.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-09T21:32:06.685 INFO:teuthology.orchestra.run.smithi090.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-09T21:32:06.686 INFO:teuthology.orchestra.run.smithi090.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-09T21:32:06.686 INFO:teuthology.orchestra.run.smithi090.stdout:hardware devices. 2024-02-09T21:32:06.686 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:32:06.686 INFO:teuthology.orchestra.run.smithi090.stdout:This subpackage provides tools for support of all platforms. 2024-02-09T21:32:06.687 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-09T21:32:06.687 INFO:teuthology.task.kernel:Updating grub on smithi090 to boot 5.14.0-419.el9.x86_64 2024-02-09T21:32:06.687 DEBUG:teuthology.orchestra.run.smithi090:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-09T21:32:07.304 INFO:teuthology.orchestra.run.smithi090.stderr:Generating grub configuration file ... 2024-02-09T21:32:09.154 INFO:teuthology.orchestra.run.smithi090.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-09T21:32:09.175 INFO:teuthology.orchestra.run.smithi090.stderr:done 2024-02-09T21:32:09.177 DEBUG:teuthology.orchestra.run.smithi090:> mktemp 2024-02-09T21:32:09.223 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/tmp.GpmFJFqiJN 2024-02-09T21:32:09.223 DEBUG:teuthology.orchestra.run.smithi090:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.GpmFJFqiJN 2024-02-09T21:32:09.251 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0666 /tmp/tmp.GpmFJFqiJN 2024-02-09T21:32:09.407 DEBUG:teuthology.orchestra.remote:smithi090:/tmp/tmp.GpmFJFqiJN is 6KB 2024-02-09T21:32:09.455 DEBUG:teuthology.orchestra.run.smithi090:> rm -fr /tmp/tmp.GpmFJFqiJN 2024-02-09T21:32:09.470 DEBUG:teuthology.orchestra.run.smithi090:> sudo /bin/ls /boot/loader/entries || true 2024-02-09T21:32:09.534 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-09T21:32:09.534 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-09T21:32:09.534 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-09T21:32:09.534 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-09T21:32:09.536 DEBUG:teuthology.orchestra.run.smithi090:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-09T21:32:09.679 DEBUG:teuthology.orchestra.run.smithi090:> sudo shutdown -r now 2024-02-09T21:32:39.683 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-02-09T21:32:39.684 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T21:32:39.685 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T21:32:58.044 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T21:33:07.045 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T21:33:07.046 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T21:33:10.112 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T21:33:22.113 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T21:33:22.114 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T21:33:25.180 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T21:33:40.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T21:33:40.182 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T21:34:12.244 DEBUG:teuthology.orchestra.run.smithi090:> true 2024-02-09T21:34:12.714 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T21:34:12.714 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-419.el9.x86_64"... 2024-02-09T21:34:12.715 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2024-02-09T21:34:12.728 INFO:teuthology.orchestra.run.smithi090.stdout:5.14.0-419.el9.x86_64 2024-02-09T21:34:12.729 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-09T21:34:12.729 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-09T21:34:12.729 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-09T21:34:13.729 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-02-09T21:34:13.730 DEBUG:teuthology.orchestra.run.smithi090:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-09T21:34:13.882 INFO:teuthology.orchestra.run.smithi090.stdout:ttyS1 2024-02-09T21:34:13.931 DEBUG:teuthology.parallel:result is None 2024-02-09T21:37:01.193 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - AppStream 53 kB/s | 19 MB 06:02 2024-02-09T21:37:05.310 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - CRB 11 MB/s | 5.9 MB 00:00 2024-02-09T21:38:13.362 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - Extras packages 227 B/s | 15 kB 01:06 2024-02-09T21:38:21.582 INFO:teuthology.orchestra.run.smithi006.stdout:Extra Packages for Enterprise Linux 2.5 MB/s | 20 MB 00:08 2024-02-09T21:38:26.629 INFO:teuthology.orchestra.run.smithi006.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-02-09T21:38:29.045 INFO:teuthology.orchestra.run.smithi006.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-09T21:38:29.045 INFO:teuthology.orchestra.run.smithi006.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-09T21:38:29.135 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-02-09T21:38:29.141 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:38:29.141 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-02-09T21:38:29.141 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:38:29.141 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-02-09T21:38:29.141 INFO:teuthology.orchestra.run.smithi006.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-09T21:38:29.141 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-02-09T21:38:29.141 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-09T21:38:29.141 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-09T21:38:29.141 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-09T21:38:29.141 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:38:29.141 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-02-09T21:38:29.142 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:38:29.142 INFO:teuthology.orchestra.run.smithi006.stdout:Install 4 Packages 2024-02-09T21:38:29.142 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:38:29.142 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 99 M 2024-02-09T21:38:29.142 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 122 M 2024-02-09T21:38:29.142 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-02-09T21:38:29.689 INFO:teuthology.orchestra.run.smithi006.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 11 MB/s | 6.0 MB 00:00 2024-02-09T21:38:30.089 INFO:teuthology.orchestra.run.smithi006.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 22 MB/s | 21 MB 00:00 2024-02-09T21:38:31.048 INFO:teuthology.orchestra.run.smithi006.stdout:(3/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 21 MB/s | 39 MB 00:01 2024-02-09T21:38:31.998 INFO:teuthology.orchestra.run.smithi006.stdout:(4/4): kernel-modules-core-5.14.0-419.el9.x86_6 14 MB/s | 33 MB 00:02 2024-02-09T21:38:32.001 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:38:32.001 INFO:teuthology.orchestra.run.smithi006.stdout:Total 35 MB/s | 99 MB 00:02 2024-02-09T21:38:32.001 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-02-09T21:38:32.453 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-02-09T21:38:32.453 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-02-09T21:38:33.525 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-02-09T21:38:33.525 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-02-09T21:38:36.061 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-02-09T21:38:36.734 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-09T21:38:36.865 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-09T21:38:39.748 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-09T21:38:39.927 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-09T21:38:44.985 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-09T21:38:45.091 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-09T21:38:50.175 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-09T21:39:20.890 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-09T21:39:20.890 INFO:teuthology.orchestra.run.smithi006.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-09T21:39:20.890 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:39:20.960 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-09T21:39:21.376 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-09T21:39:21.377 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-09T21:39:21.377 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-09T21:39:21.377 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-09T21:39:21.815 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-09T21:39:21.815 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:39:21.815 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-02-09T21:39:21.815 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-09T21:39:21.815 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-09T21:39:21.815 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-09T21:39:21.815 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-09T21:39:21.815 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:39:21.816 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-02-09T21:39:22.301 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q kernel | sort -rV | head -n 1 2024-02-09T21:39:22.438 INFO:teuthology.orchestra.run.smithi006.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-09T21:39:22.438 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-09T21:39:22.438 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-09T21:39:22.439 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-09T21:39:22.439 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-09T21:39:22.439 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-02-09T21:39:22.439 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi006.front.sepia.ceph.com, path=None, version=distro) 2024-02-09T21:39:22.439 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q kernel | sort -rV | head -n 1 2024-02-09T21:39:22.551 INFO:teuthology.orchestra.run.smithi006.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-09T21:39:22.552 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-09T21:39:22.552 DEBUG:teuthology.orchestra.run.smithi006:> sudo rpm -qi grub2-tools 2024-02-09T21:39:22.591 INFO:teuthology.orchestra.run.smithi006.stdout:Name : grub2-tools 2024-02-09T21:39:22.592 INFO:teuthology.orchestra.run.smithi006.stdout:Epoch : 1 2024-02-09T21:39:22.592 INFO:teuthology.orchestra.run.smithi006.stdout:Version : 2.06 2024-02-09T21:39:22.592 INFO:teuthology.orchestra.run.smithi006.stdout:Release : 68.el9 2024-02-09T21:39:22.592 INFO:teuthology.orchestra.run.smithi006.stdout:Architecture: x86_64 2024-02-09T21:39:22.592 INFO:teuthology.orchestra.run.smithi006.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-09T21:39:22.592 INFO:teuthology.orchestra.run.smithi006.stdout:Group : Unspecified 2024-02-09T21:39:22.592 INFO:teuthology.orchestra.run.smithi006.stdout:Size : 8264254 2024-02-09T21:39:22.592 INFO:teuthology.orchestra.run.smithi006.stdout:License : GPLv3+ 2024-02-09T21:39:22.592 INFO:teuthology.orchestra.run.smithi006.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-09T21:39:22.592 INFO:teuthology.orchestra.run.smithi006.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-09T21:39:22.592 INFO:teuthology.orchestra.run.smithi006.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-09T21:39:22.592 INFO:teuthology.orchestra.run.smithi006.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-09T21:39:22.593 INFO:teuthology.orchestra.run.smithi006.stdout:Packager : builder@centos.org 2024-02-09T21:39:22.593 INFO:teuthology.orchestra.run.smithi006.stdout:Vendor : CentOS 2024-02-09T21:39:22.593 INFO:teuthology.orchestra.run.smithi006.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-09T21:39:22.593 INFO:teuthology.orchestra.run.smithi006.stdout:Summary : Support tools for GRUB. 2024-02-09T21:39:22.593 INFO:teuthology.orchestra.run.smithi006.stdout:Description : 2024-02-09T21:39:22.593 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:39:22.593 INFO:teuthology.orchestra.run.smithi006.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-09T21:39:22.593 INFO:teuthology.orchestra.run.smithi006.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-09T21:39:22.593 INFO:teuthology.orchestra.run.smithi006.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-09T21:39:22.593 INFO:teuthology.orchestra.run.smithi006.stdout:hardware devices. 2024-02-09T21:39:22.593 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:39:22.593 INFO:teuthology.orchestra.run.smithi006.stdout:This subpackage provides tools for support of all platforms. 2024-02-09T21:39:22.594 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-09T21:39:22.594 INFO:teuthology.task.kernel:Updating grub on smithi006 to boot 5.14.0-419.el9.x86_64 2024-02-09T21:39:22.594 DEBUG:teuthology.orchestra.run.smithi006:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-09T21:39:23.353 INFO:teuthology.orchestra.run.smithi006.stderr:Generating grub configuration file ... 2024-02-09T21:39:24.785 INFO:teuthology.orchestra.run.smithi006.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-09T21:39:24.826 INFO:teuthology.orchestra.run.smithi006.stderr:done 2024-02-09T21:39:24.828 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2024-02-09T21:39:24.844 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.QecqnO5gNI 2024-02-09T21:39:24.844 DEBUG:teuthology.orchestra.run.smithi006:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.QecqnO5gNI 2024-02-09T21:39:24.922 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.QecqnO5gNI 2024-02-09T21:39:25.070 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.QecqnO5gNI is 6KB 2024-02-09T21:39:25.120 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.QecqnO5gNI 2024-02-09T21:39:25.134 DEBUG:teuthology.orchestra.run.smithi006:> sudo /bin/ls /boot/loader/entries || true 2024-02-09T21:39:25.202 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-09T21:39:25.202 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-09T21:39:25.202 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-09T21:39:25.203 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-09T21:39:25.204 DEBUG:teuthology.orchestra.run.smithi006:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-09T21:39:25.377 DEBUG:teuthology.orchestra.run.smithi006:> sudo shutdown -r now 2024-02-09T21:39:55.380 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-02-09T21:39:55.380 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-02-09T21:39:55.381 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T21:40:13.760 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.6 2024-02-09T21:40:22.762 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-02-09T21:40:22.763 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T21:41:22.766 DEBUG:teuthology.orchestra.remote:timed out 2024-02-09T21:41:34.767 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-02-09T21:41:34.769 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T21:41:35.094 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-02-09T21:41:35.519 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-02-09T21:41:35.520 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-419.el9.x86_64"... 2024-02-09T21:41:35.520 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-02-09T21:41:35.536 INFO:teuthology.orchestra.run.smithi006.stdout:5.14.0-419.el9.x86_64 2024-02-09T21:41:35.536 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-09T21:41:35.536 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-09T21:41:35.536 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-09T21:41:36.537 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-02-09T21:41:36.538 DEBUG:teuthology.orchestra.run.smithi006:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-09T21:41:36.620 INFO:teuthology.orchestra.run.smithi006.stdout:ttyS1 2024-02-09T21:41:36.654 DEBUG:teuthology.parallel:result is None 2024-02-09T21:41:36.654 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-09T21:41:36.663 INFO:teuthology.task.internal:Creating test directory... 2024-02-09T21:41:36.664 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-09T21:41:36.667 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-09T21:41:36.685 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-09T21:41:36.701 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-09T21:41:36.737 INFO:teuthology.task.internal:Creating archive directory... 2024-02-09T21:41:36.738 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-09T21:41:36.741 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-09T21:41:36.771 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-09T21:41:36.777 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-09T21:41:36.777 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-09T21:41:36.801 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-09T21:41:36.840 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T21:41:36.843 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T21:41:36.854 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T21:41:36.857 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T21:41:36.859 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-09T21:41:36.866 INFO:teuthology.task.internal:Configuring sudo... 2024-02-09T21:41:36.866 DEBUG:teuthology.orchestra.run.smithi006:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-09T21:41:36.869 DEBUG:teuthology.orchestra.run.smithi090:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-09T21:41:36.930 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-09T21:41:36.938 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-09T21:41:36.938 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-09T21:41:36.941 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-09T21:41:36.985 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-09T21:41:37.040 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-09T21:41:37.082 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-09T21:41:37.165 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-09T21:41:37.207 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-02-09T21:41:37.207 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-09T21:41:37.273 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-09T21:41:37.322 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-09T21:41:37.372 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-09T21:41:37.457 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-09T21:41:37.501 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-09T21:41:37.501 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-09T21:41:37.572 DEBUG:teuthology.orchestra.run.smithi006:> sudo service rsyslog restart 2024-02-09T21:41:37.575 DEBUG:teuthology.orchestra.run.smithi090:> sudo service rsyslog restart 2024-02-09T21:41:37.633 INFO:teuthology.orchestra.run.smithi006.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-09T21:41:37.671 INFO:teuthology.orchestra.run.smithi090.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-09T21:41:38.013 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-09T21:41:38.021 INFO:teuthology.task.internal:Starting timer... 2024-02-09T21:41:38.021 INFO:teuthology.run_tasks:Running task pcp... 2024-02-09T21:41:38.057 INFO:teuthology.run_tasks:Running task selinux... 2024-02-09T21:41:38.087 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-09T21:41:38.087 DEBUG:teuthology.orchestra.run.smithi006:> sudo service auditd rotate 2024-02-09T21:41:38.159 INFO:teuthology.orchestra.run.smithi006.stdout:Rotating logs: 2024-02-09T21:41:38.161 DEBUG:teuthology.orchestra.run.smithi090:> sudo service auditd rotate 2024-02-09T21:41:38.242 INFO:teuthology.orchestra.run.smithi090.stdout:Rotating logs: 2024-02-09T21:41:38.243 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-09T21:41:38.244 DEBUG:teuthology.orchestra.run.smithi006:> /usr/sbin/getenforce 2024-02-09T21:41:38.266 INFO:teuthology.orchestra.run.smithi006.stdout:Permissive 2024-02-09T21:41:38.267 DEBUG:teuthology.orchestra.run.smithi090:> /usr/sbin/getenforce 2024-02-09T21:41:38.294 INFO:teuthology.orchestra.run.smithi090.stdout:Permissive 2024-02-09T21:41:38.294 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi006.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi090.front.sepia.ceph.com': 'permissive'} 2024-02-09T21:41:38.294 DEBUG:teuthology.orchestra.run.smithi006:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-09T21:41:38.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T21:41:38.334 DEBUG:teuthology.orchestra.run.smithi090:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-09T21:41:38.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T21:41:38.364 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-09T21:41:38.364 DEBUG:teuthology.orchestra.run.smithi006:> sudo /usr/sbin/setenforce permissive 2024-02-09T21:41:38.409 DEBUG:teuthology.orchestra.run.smithi090:> sudo /usr/sbin/setenforce permissive 2024-02-09T21:41:38.444 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-09T21:41:38.457 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-09T21:41:38.479 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-09T21:41:38.480 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi006.front.sepia.ceph.com,smithi090.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-09T21:45:56.214 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi006.front.sepia.ceph.com'), Remote(name='ubuntu@smithi090.front.sepia.ceph.com')] 2024-02-09T21:45:56.215 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-02-09T21:45:56.217 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T21:45:56.310 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-02-09T21:45:56.387 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-02-09T21:45:56.387 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T21:45:56.388 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T21:45:56.478 DEBUG:teuthology.orchestra.run.smithi090:> true 2024-02-09T21:45:56.559 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T21:45:56.560 INFO:teuthology.run_tasks:Running task clock... 2024-02-09T21:45:56.601 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-09T21:45:56.601 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-09T21:45:56.602 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-09T21:45:56.605 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-09T21:45:56.605 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-09T21:45:56.636 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-09T21:45:56.636 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-09T21:45:56.654 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-09T21:45:56.654 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-09T21:45:56.682 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: ntpd: command not found 2024-02-09T21:45:56.687 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: ntpd: command not found 2024-02-09T21:45:56.697 INFO:teuthology.orchestra.run.smithi006.stdout:506 Cannot talk to daemon 2024-02-09T21:45:56.701 INFO:teuthology.orchestra.run.smithi090.stdout:506 Cannot talk to daemon 2024-02-09T21:45:56.714 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-09T21:45:56.718 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-09T21:45:56.730 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-09T21:45:56.734 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-09T21:45:56.782 INFO:teuthology.orchestra.run.smithi006.stderr:bash: line 1: ntpq: command not found 2024-02-09T21:45:56.785 INFO:teuthology.orchestra.run.smithi006.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-09T21:45:56.785 INFO:teuthology.orchestra.run.smithi006.stdout:=============================================================================== 2024-02-09T21:45:56.785 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T21:45:56.785 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T21:45:56.785 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T21:45:56.785 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T21:45:56.788 INFO:teuthology.orchestra.run.smithi090.stderr:bash: line 1: ntpq: command not found 2024-02-09T21:45:56.792 INFO:teuthology.orchestra.run.smithi090.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-09T21:45:56.792 INFO:teuthology.orchestra.run.smithi090.stdout:=============================================================================== 2024-02-09T21:45:56.792 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T21:45:56.792 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T21:45:56.792 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T21:45:56.792 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T21:45:56.792 INFO:teuthology.run_tasks:Running task install... 2024-02-09T21:45:56.801 DEBUG:teuthology.task.install:project ceph 2024-02-09T21:45:56.802 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'}} 2024-02-09T21:45:56.802 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'} 2024-02-09T21:45:56.802 INFO:teuthology.task.install:Using flavor: default 2024-02-09T21:45:56.808 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-09T21:45:56.809 INFO:teuthology.task.install:extra packages: [] 2024-02-09T21:45:56.809 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', 'tag': None, 'wait_for_package': False} 2024-02-09T21:45:56.810 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-09T21:45:56.810 INFO:teuthology.packaging:ref: None 2024-02-09T21:45:56.810 INFO:teuthology.packaging:tag: None 2024-02-09T21:45:56.810 INFO:teuthology.packaging:branch: quincy 2024-02-09T21:45:56.810 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T21:45:56.810 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-09T21:45:56.813 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', 'tag': None, 'wait_for_package': False} 2024-02-09T21:45:56.813 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-09T21:45:56.813 INFO:teuthology.packaging:ref: None 2024-02-09T21:45:56.813 INFO:teuthology.packaging:tag: None 2024-02-09T21:45:56.813 INFO:teuthology.packaging:branch: quincy 2024-02-09T21:45:56.813 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T21:45:56.813 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-09T21:45:57.050 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/quincy/72ce7c1d1d5a2a1807db94c08aa742a26968c21f/centos/9/flavors/default/ 2024-02-09T21:45:57.050 INFO:teuthology.task.install.rpm:Package version is 17.2.7-505.g72ce7c1d 2024-02-09T21:45:57.074 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/quincy/72ce7c1d1d5a2a1807db94c08aa742a26968c21f/centos/9/flavors/default/ 2024-02-09T21:45:57.074 INFO:teuthology.task.install.rpm:Package version is 17.2.7-505.g72ce7c1d 2024-02-09T21:45:57.185 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/quincy/72ce7c1d1d5a2a1807db94c08aa742a26968c21f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/72ce7c1d1d5a2a1807db94c08aa742a26968c21f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/72ce7c1d1d5a2a1807db94c08aa742a26968c21f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-09T21:45:57.185 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-02-09T21:45:57.185 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-09T21:45:57.213 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-09T21:45:57.213 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-09T21:45:57.213 INFO:teuthology.packaging:ref: None 2024-02-09T21:45:57.213 INFO:teuthology.packaging:tag: None 2024-02-09T21:45:57.214 INFO:teuthology.packaging:branch: quincy 2024-02-09T21:45:57.214 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T21:45:57.214 DEBUG:teuthology.orchestra.run.smithi006:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-09T21:45:57.216 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/quincy/72ce7c1d1d5a2a1807db94c08aa742a26968c21f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/72ce7c1d1d5a2a1807db94c08aa742a26968c21f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/72ce7c1d1d5a2a1807db94c08aa742a26968c21f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-09T21:45:57.216 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-09T21:45:57.217 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-09T21:45:57.245 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-09T21:45:57.245 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-09T21:45:57.245 INFO:teuthology.packaging:ref: None 2024-02-09T21:45:57.245 INFO:teuthology.packaging:tag: None 2024-02-09T21:45:57.245 INFO:teuthology.packaging:branch: quincy 2024-02-09T21:45:57.245 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T21:45:57.245 DEBUG:teuthology.orchestra.run.smithi090:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-09T21:45:57.280 DEBUG:teuthology.orchestra.run.smithi006:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-09T21:45:57.313 DEBUG:teuthology.orchestra.run.smithi090:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-09T21:45:57.361 DEBUG:teuthology.orchestra.run.smithi006:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-09T21:45:57.394 DEBUG:teuthology.orchestra.run.smithi090:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-09T21:45:57.426 INFO:teuthology.orchestra.run.smithi006.stdout:check_obsoletes = 1 2024-02-09T21:45:57.428 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum clean all 2024-02-09T21:45:57.462 INFO:teuthology.orchestra.run.smithi090.stdout:check_obsoletes = 1 2024-02-09T21:45:57.463 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum clean all 2024-02-09T21:45:57.683 INFO:teuthology.orchestra.run.smithi006.stdout:74 files removed 2024-02-09T21:45:57.707 INFO:teuthology.orchestra.run.smithi090.stdout:74 files removed 2024-02-09T21:45:57.710 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-radosgw 2024-02-09T21:45:57.734 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-radosgw 2024-02-09T21:45:58.498 INFO:teuthology.orchestra.run.smithi006.stdout:ceph packages for x86_64 167 kB/s | 77 kB 00:00 2024-02-09T21:45:58.515 INFO:teuthology.orchestra.run.smithi090.stdout:ceph packages for x86_64 167 kB/s | 77 kB 00:00 2024-02-09T21:45:58.882 INFO:teuthology.orchestra.run.smithi090.stdout:ceph noarch packages 29 kB/s | 9.1 kB 00:00 2024-02-09T21:45:58.890 INFO:teuthology.orchestra.run.smithi006.stdout:ceph noarch packages 27 kB/s | 9.1 kB 00:00 2024-02-09T21:45:59.224 INFO:teuthology.orchestra.run.smithi090.stdout:ceph source packages 5.9 kB/s | 1.8 kB 00:00 2024-02-09T21:45:59.374 INFO:teuthology.orchestra.run.smithi006.stdout:ceph source packages 4.0 kB/s | 1.8 kB 00:00 2024-02-09T21:45:59.644 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-02-09T21:45:59.835 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-02-09T21:46:01.053 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-02-09T21:46:01.419 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - BaseOS 5.0 MB/s | 2.0 MB 00:00 2024-02-09T21:46:02.712 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - BaseOS 8.4 MB/s | 8.0 MB 00:00 2024-02-09T21:46:03.353 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - BaseOS 4.0 MB/s | 8.0 MB 00:01 2024-02-09T21:46:05.363 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-02-09T21:46:06.877 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - AppStream 6.2 MB/s | 19 MB 00:03 2024-02-09T21:46:10.049 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - CRB 4.9 MB/s | 5.9 MB 00:01 2024-02-09T21:46:11.892 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - Extras packages 21 kB/s | 15 kB 00:00 2024-02-09T21:46:12.478 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 5.9 MB 00:02 2024-02-09T21:46:13.428 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 15 MB/s | 20 MB 00:01 2024-02-09T21:46:13.921 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 15 kB 00:00 2024-02-09T21:46:17.561 INFO:teuthology.orchestra.run.smithi006.stdout:Extra Packages for Enterprise Linux 5.8 MB/s | 20 MB 00:03 2024-02-09T21:46:18.370 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 20 kB/s | 1.7 kB 00:00 2024-02-09T21:46:21.089 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-09T21:46:21.100 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:46:21.100 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-02-09T21:46:21.100 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:46:21.100 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-09T21:46:21.101 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 14 M 2024-02-09T21:46:21.101 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrading: 2024-02-09T21:46:21.101 INFO:teuthology.orchestra.run.smithi090.stdout: librados2 x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 3.1 M 2024-02-09T21:46:21.101 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1 x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 2.9 M 2024-02-09T21:46:21.101 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-02-09T21:46:21.101 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 5.0 M 2024-02-09T21:46:21.101 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 23 M 2024-02-09T21:46:21.101 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 25 k 2024-02-09T21:46:21.101 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-09T21:46:21.101 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2 x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 635 k 2024-02-09T21:46:21.101 INFO:teuthology.orchestra.run.smithi090.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-09T21:46:21.102 INFO:teuthology.orchestra.run.smithi090.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-09T21:46:21.102 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-09T21:46:21.102 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1 x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 458 k 2024-02-09T21:46:21.102 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-09T21:46:21.102 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2 x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 4.9 M 2024-02-09T21:46:21.102 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-09T21:46:21.102 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-09T21:46:21.102 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 45 k 2024-02-09T21:46:21.102 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 116 k 2024-02-09T21:46:21.102 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 156 k 2024-02-09T21:46:21.102 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-09T21:46:21.102 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 320 k 2024-02-09T21:46:21.102 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 293 k 2024-02-09T21:46:21.103 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 99 k 2024-02-09T21:46:21.103 INFO:teuthology.orchestra.run.smithi090.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-09T21:46:21.103 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:46:21.103 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-09T21:46:21.103 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:46:21.103 INFO:teuthology.orchestra.run.smithi090.stdout:Install 22 Packages 2024-02-09T21:46:21.103 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrade 2 Packages 2024-02-09T21:46:21.103 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:46:21.104 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 58 M 2024-02-09T21:46:21.104 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-09T21:46:22.605 INFO:teuthology.orchestra.run.smithi006.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-02-09T21:46:23.347 INFO:teuthology.orchestra.run.smithi090.stdout:(1/24): ceph-base-17.2.7-505.g72ce7c1d.el9.x86_ 2.5 MB/s | 5.0 MB 00:02 2024-02-09T21:46:23.405 INFO:teuthology.orchestra.run.smithi090.stdout:(2/24): ceph-selinux-17.2.7-505.g72ce7c1d.el9.x 432 kB/s | 25 kB 00:00 2024-02-09T21:46:23.656 INFO:teuthology.orchestra.run.smithi090.stdout:(3/24): libcephfs2-17.2.7-505.g72ce7c1d.el9.x86 2.5 MB/s | 635 kB 00:00 2024-02-09T21:46:23.831 INFO:teuthology.orchestra.run.smithi090.stdout:(4/24): libradosstriper1-17.2.7-505.g72ce7c1d.e 2.6 MB/s | 458 kB 00:00 2024-02-09T21:46:25.123 INFO:teuthology.orchestra.run.smithi090.stdout:(5/24): librgw2-17.2.7-505.g72ce7c1d.el9.x86_64 3.8 MB/s | 4.9 MB 00:01 2024-02-09T21:46:25.315 INFO:teuthology.orchestra.run.smithi090.stdout:(6/24): ceph-radosgw-17.2.7-505.g72ce7c1d.el9.x 3.5 MB/s | 14 MB 00:04 2024-02-09T21:46:25.340 INFO:teuthology.orchestra.run.smithi090.stdout:(7/24): python3-ceph-argparse-17.2.7-505.g72ce7 208 kB/s | 45 kB 00:00 2024-02-09T21:46:25.391 INFO:teuthology.orchestra.run.smithi090.stdout:(8/24): python3-ceph-common-17.2.7-505.g72ce7c1 1.5 MB/s | 116 kB 00:00 2024-02-09T21:46:25.393 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-02-09T21:46:25.408 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:46:25.408 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-02-09T21:46:25.408 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:46:25.408 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-02-09T21:46:25.408 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 14 M 2024-02-09T21:46:25.409 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrading: 2024-02-09T21:46:25.409 INFO:teuthology.orchestra.run.smithi006.stdout: librados2 x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 3.1 M 2024-02-09T21:46:25.409 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1 x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 2.9 M 2024-02-09T21:46:25.409 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-02-09T21:46:25.409 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 5.0 M 2024-02-09T21:46:25.409 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 23 M 2024-02-09T21:46:25.409 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-selinux x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 25 k 2024-02-09T21:46:25.409 INFO:teuthology.orchestra.run.smithi006.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-09T21:46:25.409 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2 x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 635 k 2024-02-09T21:46:25.409 INFO:teuthology.orchestra.run.smithi006.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-09T21:46:25.409 INFO:teuthology.orchestra.run.smithi006.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-09T21:46:25.409 INFO:teuthology.orchestra.run.smithi006.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-09T21:46:25.409 INFO:teuthology.orchestra.run.smithi006.stdout: libradosstriper1 x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 458 k 2024-02-09T21:46:25.409 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-09T21:46:25.409 INFO:teuthology.orchestra.run.smithi006.stdout: librgw2 x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 4.9 M 2024-02-09T21:46:25.410 INFO:teuthology.orchestra.run.smithi006.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-09T21:46:25.410 INFO:teuthology.orchestra.run.smithi006.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-09T21:46:25.410 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 45 k 2024-02-09T21:46:25.410 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 116 k 2024-02-09T21:46:25.410 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 156 k 2024-02-09T21:46:25.410 INFO:teuthology.orchestra.run.smithi006.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-09T21:46:25.410 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 320 k 2024-02-09T21:46:25.410 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 293 k 2024-02-09T21:46:25.410 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 99 k 2024-02-09T21:46:25.410 INFO:teuthology.orchestra.run.smithi006.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-09T21:46:25.410 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:46:25.410 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-02-09T21:46:25.410 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:46:25.410 INFO:teuthology.orchestra.run.smithi006.stdout:Install 22 Packages 2024-02-09T21:46:25.411 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrade 2 Packages 2024-02-09T21:46:25.411 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:46:25.411 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 58 M 2024-02-09T21:46:25.411 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-02-09T21:46:25.441 INFO:teuthology.orchestra.run.smithi090.stdout:(9/24): python3-cephfs-17.2.7-505.g72ce7c1d.el9 1.5 MB/s | 156 kB 00:00 2024-02-09T21:46:25.525 INFO:teuthology.orchestra.run.smithi090.stdout:(10/24): python3-rados-17.2.7-505.g72ce7c1d.el9 2.3 MB/s | 320 kB 00:00 2024-02-09T21:46:25.600 INFO:teuthology.orchestra.run.smithi090.stdout:(11/24): python3-rbd-17.2.7-505.g72ce7c1d.el9.x 1.8 MB/s | 293 kB 00:00 2024-02-09T21:46:25.634 INFO:teuthology.orchestra.run.smithi090.stdout:(12/24): python3-rgw-17.2.7-505.g72ce7c1d.el9.x 914 kB/s | 99 kB 00:00 2024-02-09T21:46:25.751 INFO:teuthology.orchestra.run.smithi090.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 479 kB/s | 44 kB 00:00 2024-02-09T21:46:25.776 INFO:teuthology.orchestra.run.smithi090.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 907 kB/s | 159 kB 00:00 2024-02-09T21:46:25.835 INFO:teuthology.orchestra.run.smithi090.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-02-09T21:46:25.869 INFO:teuthology.orchestra.run.smithi090.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-02-09T21:46:25.895 INFO:teuthology.orchestra.run.smithi090.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 693 kB/s | 41 kB 00:00 2024-02-09T21:46:26.181 INFO:teuthology.orchestra.run.smithi006.stdout:(1/24): ceph-base-17.2.7-505.g72ce7c1d.el9.x86_ 8.2 MB/s | 5.0 MB 00:00 2024-02-09T21:46:26.232 INFO:teuthology.orchestra.run.smithi006.stdout:(2/24): ceph-selinux-17.2.7-505.g72ce7c1d.el9.x 503 kB/s | 25 kB 00:00 2024-02-09T21:46:26.315 INFO:teuthology.orchestra.run.smithi006.stdout:(3/24): libcephfs2-17.2.7-505.g72ce7c1d.el9.x86 7.5 MB/s | 635 kB 00:00 2024-02-09T21:46:26.387 INFO:teuthology.orchestra.run.smithi090.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 99 kB/s | 49 kB 00:00 2024-02-09T21:46:26.390 INFO:teuthology.orchestra.run.smithi006.stdout:(4/24): libradosstriper1-17.2.7-505.g72ce7c1d.e 6.0 MB/s | 458 kB 00:00 2024-02-09T21:46:26.554 INFO:teuthology.orchestra.run.smithi090.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 404 kB/s | 67 kB 00:00 2024-02-09T21:46:26.686 INFO:teuthology.orchestra.run.smithi090.stdout:(20/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 430 kB/s | 309 kB 00:00 2024-02-09T21:46:26.799 INFO:teuthology.orchestra.run.smithi006.stdout:(5/24): librgw2-17.2.7-505.g72ce7c1d.el9.x86_64 12 MB/s | 4.9 MB 00:00 2024-02-09T21:46:26.858 INFO:teuthology.orchestra.run.smithi006.stdout:(6/24): python3-ceph-argparse-17.2.7-505.g72ce7 775 kB/s | 45 kB 00:00 2024-02-09T21:46:26.935 INFO:teuthology.orchestra.run.smithi006.stdout:(7/24): python3-ceph-common-17.2.7-505.g72ce7c1 1.5 MB/s | 116 kB 00:00 2024-02-09T21:46:27.035 INFO:teuthology.orchestra.run.smithi006.stdout:(8/24): python3-cephfs-17.2.7-505.g72ce7c1d.el9 1.5 MB/s | 156 kB 00:00 2024-02-09T21:46:27.088 INFO:teuthology.orchestra.run.smithi090.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 3.0 MB/s | 1.6 MB 00:00 2024-02-09T21:46:27.127 INFO:teuthology.orchestra.run.smithi006.stdout:(9/24): python3-rados-17.2.7-505.g72ce7c1d.el9. 3.4 MB/s | 320 kB 00:00 2024-02-09T21:46:27.228 INFO:teuthology.orchestra.run.smithi006.stdout:(10/24): python3-rbd-17.2.7-505.g72ce7c1d.el9.x 2.9 MB/s | 293 kB 00:00 2024-02-09T21:46:27.328 INFO:teuthology.orchestra.run.smithi006.stdout:(11/24): python3-rgw-17.2.7-505.g72ce7c1d.el9.x 995 kB/s | 99 kB 00:00 2024-02-09T21:46:27.445 INFO:teuthology.orchestra.run.smithi006.stdout:(12/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2024-02-09T21:46:27.487 INFO:teuthology.orchestra.run.smithi006.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-02-09T21:46:27.654 INFO:teuthology.orchestra.run.smithi006.stdout:(14/24): ceph-radosgw-17.2.7-505.g72ce7c1d.el9. 6.7 MB/s | 14 MB 00:02 2024-02-09T21:46:27.679 INFO:teuthology.orchestra.run.smithi006.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 3.4 MB/s | 661 kB 00:00 2024-02-09T21:46:27.704 INFO:teuthology.orchestra.run.smithi006.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.6 MB/s | 282 kB 00:00 2024-02-09T21:46:27.729 INFO:teuthology.orchestra.run.smithi006.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 828 kB/s | 41 kB 00:00 2024-02-09T21:46:27.872 INFO:teuthology.orchestra.run.smithi090.stdout:(22/24): ceph-common-17.2.7-505.g72ce7c1d.el9.x 3.4 MB/s | 23 MB 00:06 2024-02-09T21:46:28.030 INFO:teuthology.orchestra.run.smithi090.stdout:(23/24): librados2-17.2.7-505.g72ce7c1d.el9.x86 2.1 MB/s | 3.1 MB 00:01 2024-02-09T21:46:28.071 INFO:teuthology.orchestra.run.smithi006.stdout:(18/24): ceph-common-17.2.7-505.g72ce7c1d.el9.x 9.0 MB/s | 23 MB 00:02 2024-02-09T21:46:28.096 INFO:teuthology.orchestra.run.smithi006.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 132 kB/s | 49 kB 00:00 2024-02-09T21:46:28.198 INFO:teuthology.orchestra.run.smithi090.stdout:(24/24): librbd1-17.2.7-505.g72ce7c1d.el9.x86_6 2.6 MB/s | 2.9 MB 00:01 2024-02-09T21:46:28.204 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:46:28.204 INFO:teuthology.orchestra.run.smithi090.stdout:Total 8.1 MB/s | 58 MB 00:07 2024-02-09T21:46:28.204 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-09T21:46:28.235 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-09T21:46:28.235 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-09T21:46:28.461 INFO:teuthology.orchestra.run.smithi006.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 173 kB/s | 67 kB 00:00 2024-02-09T21:46:28.489 INFO:teuthology.orchestra.run.smithi006.stdout:(21/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 394 kB/s | 309 kB 00:00 2024-02-09T21:46:28.647 INFO:teuthology.orchestra.run.smithi006.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 2.9 MB/s | 1.6 MB 00:00 2024-02-09T21:46:28.706 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-09T21:46:28.706 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-09T21:46:28.822 INFO:teuthology.orchestra.run.smithi006.stdout:(23/24): librbd1-17.2.7-505.g72ce7c1d.el9.x86_6 8.8 MB/s | 2.9 MB 00:00 2024-02-09T21:46:28.881 INFO:teuthology.orchestra.run.smithi006.stdout:(24/24): librados2-17.2.7-505.g72ce7c1d.el9.x86 7.3 MB/s | 3.1 MB 00:00 2024-02-09T21:46:28.885 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:46:28.885 INFO:teuthology.orchestra.run.smithi006.stdout:Total 17 MB/s | 58 MB 00:03 2024-02-09T21:46:28.885 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-02-09T21:46:28.923 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-02-09T21:46:28.923 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-02-09T21:46:29.420 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-02-09T21:46:29.421 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-02-09T21:46:29.896 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-09T21:46:30.233 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-02-09T21:46:30.317 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librados2-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/26 2024-02-09T21:46:30.408 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/26 2024-02-09T21:46:30.495 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rados-2:17.2.7-505.g72ce7c1d.el9.x86_64 3/26 2024-02-09T21:46:30.527 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs2-2:17.2.7-505.g72ce7c1d.el9.x86_64 4/26 2024-02-09T21:46:30.615 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephfs2-2:17.2.7-505.g72ce7c1d.el9.x86_64 4/26 2024-02-09T21:46:30.662 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-02-09T21:46:30.716 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-02-09T21:46:30.747 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-02-09T21:46:31.094 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-02-09T21:46:31.150 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : librados2-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/26 2024-02-09T21:46:31.156 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-02-09T21:46:31.190 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librgw2-2:17.2.7-505.g72ce7c1d.el9.x86_64 8/26 2024-02-09T21:46:31.242 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados2-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/26 2024-02-09T21:46:31.252 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librgw2-2:17.2.7-505.g72ce7c1d.el9.x86_64 8/26 2024-02-09T21:46:31.291 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-02-09T21:46:31.331 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rados-2:17.2.7-505.g72ce7c1d.el9.x86_64 3/26 2024-02-09T21:46:31.332 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-argparse-2:17.2.7-505.g72ce7c1d.el9.x 10/26 2024-02-09T21:46:31.368 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephfs2-2:17.2.7-505.g72ce7c1d.el9.x86_64 4/26 2024-02-09T21:46:31.369 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cephfs-2:17.2.7-505.g72ce7c1d.el9.x86_64 11/26 2024-02-09T21:46:31.432 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 12/26 2024-02-09T21:46:31.457 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephfs2-2:17.2.7-505.g72ce7c1d.el9.x86_64 4/26 2024-02-09T21:46:31.460 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libradosstriper1-2:17.2.7-505.g72ce7c1d.el9.x86_64 13/26 2024-02-09T21:46:31.511 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-02-09T21:46:31.526 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libradosstriper1-2:17.2.7-505.g72ce7c1d.el9.x86_64 13/26 2024-02-09T21:46:31.548 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-02-09T21:46:31.579 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-02-09T21:46:31.618 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-02-09T21:46:31.652 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-02-09T21:46:31.955 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-02-09T21:46:31.991 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librbd1-2:17.2.7-505.g72ce7c1d.el9.x86_64 18/26 2024-02-09T21:46:32.006 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-02-09T21:46:32.031 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librgw2-2:17.2.7-505.g72ce7c1d.el9.x86_64 8/26 2024-02-09T21:46:32.064 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-2:17.2.7-505.g72ce7c1d.el9.x86_64 18/26 2024-02-09T21:46:32.075 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librgw2-2:17.2.7-505.g72ce7c1d.el9.x86_64 8/26 2024-02-09T21:46:32.104 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-02-09T21:46:32.137 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 19/26 2024-02-09T21:46:32.148 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ceph-argparse-2:17.2.7-505.g72ce7c1d.el9.x 10/26 2024-02-09T21:46:32.169 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86 20/26 2024-02-09T21:46:32.184 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cephfs-2:17.2.7-505.g72ce7c1d.el9.x86_64 11/26 2024-02-09T21:46:32.248 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 12/26 2024-02-09T21:46:32.274 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libradosstriper1-2:17.2.7-505.g72ce7c1d.el9.x86_64 13/26 2024-02-09T21:46:32.339 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libradosstriper1-2:17.2.7-505.g72ce7c1d.el9.x86_64 13/26 2024-02-09T21:46:32.380 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-02-09T21:46:32.424 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-02-09T21:46:32.457 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-02-09T21:46:32.765 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-02-09T21:46:32.795 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : librbd1-2:17.2.7-505.g72ce7c1d.el9.x86_64 18/26 2024-02-09T21:46:32.885 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librbd1-2:17.2.7-505.g72ce7c1d.el9.x86_64 18/26 2024-02-09T21:46:32.960 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 19/26 2024-02-09T21:46:32.988 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86 20/26 2024-02-09T21:46:34.666 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 21/26 2024-02-09T21:46:34.734 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 21/26 2024-02-09T21:46:35.318 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 21/26 2024-02-09T21:46:35.505 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-base-2:17.2.7-505.g72ce7c1d.el9.x86_64 22/26 2024-02-09T21:46:35.522 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 21/26 2024-02-09T21:46:35.569 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 21/26 2024-02-09T21:46:35.596 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-base-2:17.2.7-505.g72ce7c1d.el9.x86_64 22/26 2024-02-09T21:46:35.596 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-09T21:46:35.596 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-09T21:46:35.596 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:46:35.640 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-selinux-2:17.2.7-505.g72ce7c1d.el9.x86_64 23/26 2024-02-09T21:46:36.144 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 21/26 2024-02-09T21:46:36.180 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-base-2:17.2.7-505.g72ce7c1d.el9.x86_64 22/26 2024-02-09T21:46:36.271 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-base-2:17.2.7-505.g72ce7c1d.el9.x86_64 22/26 2024-02-09T21:46:36.271 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-09T21:46:36.271 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-09T21:46:36.271 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:46:36.315 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-selinux-2:17.2.7-505.g72ce7c1d.el9.x86_64 23/26 2024-02-09T21:46:45.517 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-selinux-2:17.2.7-505.g72ce7c1d.el9.x86_64 23/26 2024-02-09T21:46:45.517 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /sys 2024-02-09T21:46:45.517 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /proc 2024-02-09T21:46:45.518 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /mnt 2024-02-09T21:46:45.518 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /var/tmp 2024-02-09T21:46:45.518 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /home 2024-02-09T21:46:45.518 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /root 2024-02-09T21:46:45.518 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /tmp 2024-02-09T21:46:45.518 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:46:46.433 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-selinux-2:17.2.7-505.g72ce7c1d.el9.x86_64 23/26 2024-02-09T21:46:46.434 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /sys 2024-02-09T21:46:46.434 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /proc 2024-02-09T21:46:46.434 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /mnt 2024-02-09T21:46:46.434 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /var/tmp 2024-02-09T21:46:46.434 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /home 2024-02-09T21:46:46.434 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /root 2024-02-09T21:46:46.434 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /tmp 2024-02-09T21:46:46.434 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:46:46.836 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-radosgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 24/26 2024-02-09T21:46:46.914 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 24/26 2024-02-09T21:46:46.914 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T21:46:46.914 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-09T21:46:46.914 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-09T21:46:46.915 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-09T21:46:46.915 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:46:46.916 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-09T21:46:47.006 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-09T21:46:47.006 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-09T21:46:47.749 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-radosgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 24/26 2024-02-09T21:46:47.810 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 24/26 2024-02-09T21:46:47.810 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T21:46:47.810 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-09T21:46:47.811 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-09T21:46:47.811 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-09T21:46:47.811 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:46:47.812 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-09T21:46:47.894 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-09T21:46:47.894 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-09T21:46:48.482 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-09T21:46:48.482 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-base-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/26 2024-02-09T21:46:48.482 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/26 2024-02-09T21:46:48.483 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-radosgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 3/26 2024-02-09T21:46:48.483 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-selinux-2:17.2.7-505.g72ce7c1d.el9.x86_64 4/26 2024-02-09T21:46:48.483 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs2-2:17.2.7-505.g72ce7c1d.el9.x86_64 5/26 2024-02-09T21:46:48.483 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libradosstriper1-2:17.2.7-505.g72ce7c1d.el9.x86_64 6/26 2024-02-09T21:46:48.483 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librgw2-2:17.2.7-505.g72ce7c1d.el9.x86_64 7/26 2024-02-09T21:46:48.483 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-argparse-2:17.2.7-505.g72ce7c1d.el9.x 8/26 2024-02-09T21:46:48.483 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86 9/26 2024-02-09T21:46:48.483 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cephfs-2:17.2.7-505.g72ce7c1d.el9.x86_64 10/26 2024-02-09T21:46:48.483 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rados-2:17.2.7-505.g72ce7c1d.el9.x86_64 11/26 2024-02-09T21:46:48.483 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 12/26 2024-02-09T21:46:48.483 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 13/26 2024-02-09T21:46:48.483 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-02-09T21:46:48.483 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-02-09T21:46:48.483 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-02-09T21:46:48.483 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-02-09T21:46:48.484 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-02-09T21:46:48.484 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-02-09T21:46:48.484 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-02-09T21:46:48.484 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-02-09T21:46:48.484 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-02-09T21:46:48.484 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-2:17.2.7-505.g72ce7c1d.el9.x86_64 23/26 2024-02-09T21:46:48.484 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-02-09T21:46:48.484 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-2:17.2.7-505.g72ce7c1d.el9.x86_64 25/26 2024-02-09T21:46:49.134 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-09T21:46:49.134 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-base-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/26 2024-02-09T21:46:49.134 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/26 2024-02-09T21:46:49.134 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-radosgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 3/26 2024-02-09T21:46:49.134 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-selinux-2:17.2.7-505.g72ce7c1d.el9.x86_64 4/26 2024-02-09T21:46:49.134 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephfs2-2:17.2.7-505.g72ce7c1d.el9.x86_64 5/26 2024-02-09T21:46:49.134 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libradosstriper1-2:17.2.7-505.g72ce7c1d.el9.x86_64 6/26 2024-02-09T21:46:49.135 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librgw2-2:17.2.7-505.g72ce7c1d.el9.x86_64 7/26 2024-02-09T21:46:49.135 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ceph-argparse-2:17.2.7-505.g72ce7c1d.el9.x 8/26 2024-02-09T21:46:49.135 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86 9/26 2024-02-09T21:46:49.135 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cephfs-2:17.2.7-505.g72ce7c1d.el9.x86_64 10/26 2024-02-09T21:46:49.135 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rados-2:17.2.7-505.g72ce7c1d.el9.x86_64 11/26 2024-02-09T21:46:49.135 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 12/26 2024-02-09T21:46:49.135 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 13/26 2024-02-09T21:46:49.135 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-02-09T21:46:49.135 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-02-09T21:46:49.135 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-02-09T21:46:49.135 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-02-09T21:46:49.135 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-02-09T21:46:49.135 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-02-09T21:46:49.135 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-02-09T21:46:49.136 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-02-09T21:46:49.136 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-02-09T21:46:49.136 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados2-2:17.2.7-505.g72ce7c1d.el9.x86_64 23/26 2024-02-09T21:46:49.136 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-02-09T21:46:49.136 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd1-2:17.2.7-505.g72ce7c1d.el9.x86_64 25/26 2024-02-09T21:46:49.539 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-02-09T21:46:49.539 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:46:49.539 INFO:teuthology.orchestra.run.smithi090.stdout:Upgraded: 2024-02-09T21:46:49.539 INFO:teuthology.orchestra.run.smithi090.stdout: librados2-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:46:49.539 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:46:49.539 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-09T21:46:49.539 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:46:49.539 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:46:49.539 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:46:49.539 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:46:49.539 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-09T21:46:49.539 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:46:49.540 INFO:teuthology.orchestra.run.smithi090.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-09T21:46:49.540 INFO:teuthology.orchestra.run.smithi090.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-09T21:46:49.540 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-09T21:46:49.540 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:46:49.540 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-09T21:46:49.540 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:46:49.540 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-09T21:46:49.540 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-09T21:46:49.540 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:46:49.540 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:46:49.540 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:46:49.540 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-09T21:46:49.540 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:46:49.540 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:46:49.541 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:46:49.541 INFO:teuthology.orchestra.run.smithi090.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-09T21:46:49.541 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:46:49.541 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-09T21:46:49.947 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-test 2024-02-09T21:46:49.958 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-02-09T21:46:49.959 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:46:49.959 INFO:teuthology.orchestra.run.smithi006.stdout:Upgraded: 2024-02-09T21:46:49.959 INFO:teuthology.orchestra.run.smithi006.stdout: librados2-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:46:49.959 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:46:49.959 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-02-09T21:46:49.959 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:46:49.959 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:46:49.959 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:46:49.959 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-selinux-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:46:49.959 INFO:teuthology.orchestra.run.smithi006.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-09T21:46:49.959 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:46:49.959 INFO:teuthology.orchestra.run.smithi006.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-09T21:46:49.959 INFO:teuthology.orchestra.run.smithi006.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-09T21:46:49.960 INFO:teuthology.orchestra.run.smithi006.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-09T21:46:49.960 INFO:teuthology.orchestra.run.smithi006.stdout: libradosstriper1-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:46:49.960 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-09T21:46:49.960 INFO:teuthology.orchestra.run.smithi006.stdout: librgw2-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:46:49.960 INFO:teuthology.orchestra.run.smithi006.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-09T21:46:49.960 INFO:teuthology.orchestra.run.smithi006.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-09T21:46:49.960 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:46:49.960 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:46:49.960 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:46:49.960 INFO:teuthology.orchestra.run.smithi006.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-09T21:46:49.960 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:46:49.960 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:46:49.960 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:46:49.960 INFO:teuthology.orchestra.run.smithi006.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-09T21:46:49.961 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:46:49.961 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-02-09T21:46:50.342 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-test 2024-02-09T21:46:50.533 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:32 ago on Fri 09 Feb 2024 09:46:18 PM UTC. 2024-02-09T21:46:50.732 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-09T21:46:50.732 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:46:50.732 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-02-09T21:46:50.733 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:46:50.733 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-09T21:46:50.733 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 48 M 2024-02-09T21:46:50.733 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-02-09T21:46:50.733 INFO:teuthology.orchestra.run.smithi090.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-09T21:46:50.733 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 162 k 2024-02-09T21:46:50.733 INFO:teuthology.orchestra.run.smithi090.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-09T21:46:50.733 INFO:teuthology.orchestra.run.smithi090.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-09T21:46:50.733 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-09T21:46:50.733 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:46:50.733 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-09T21:46:50.733 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:46:50.733 INFO:teuthology.orchestra.run.smithi090.stdout:Install 6 Packages 2024-02-09T21:46:50.733 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:46:50.734 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 49 M 2024-02-09T21:46:50.734 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 196 M 2024-02-09T21:46:50.734 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-09T21:46:50.928 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:00:28 ago on Fri 09 Feb 2024 09:46:22 PM UTC. 2024-02-09T21:46:50.965 INFO:teuthology.orchestra.run.smithi090.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.3 MB/s | 300 kB 00:00 2024-02-09T21:46:50.999 INFO:teuthology.orchestra.run.smithi090.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-02-09T21:46:51.042 INFO:teuthology.orchestra.run.smithi090.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 4.2 MB/s | 186 kB 00:00 2024-02-09T21:46:51.076 INFO:teuthology.orchestra.run.smithi090.stdout:(4/6): libcephsqlite-17.2.7-505.g72ce7c1d.el9.x 476 kB/s | 162 kB 00:00 2024-02-09T21:46:51.092 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-02-09T21:46:51.093 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:46:51.093 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-02-09T21:46:51.093 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:46:51.093 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-02-09T21:46:51.093 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 48 M 2024-02-09T21:46:51.093 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-02-09T21:46:51.093 INFO:teuthology.orchestra.run.smithi006.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-09T21:46:51.093 INFO:teuthology.orchestra.run.smithi006.stdout: libcephsqlite x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 162 k 2024-02-09T21:46:51.093 INFO:teuthology.orchestra.run.smithi006.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-09T21:46:51.093 INFO:teuthology.orchestra.run.smithi006.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-09T21:46:51.093 INFO:teuthology.orchestra.run.smithi006.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-09T21:46:51.093 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:46:51.093 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-02-09T21:46:51.094 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:46:51.094 INFO:teuthology.orchestra.run.smithi006.stdout:Install 6 Packages 2024-02-09T21:46:51.094 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:46:51.094 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 49 M 2024-02-09T21:46:51.094 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 196 M 2024-02-09T21:46:51.094 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-02-09T21:46:51.101 INFO:teuthology.orchestra.run.smithi090.stdout:(5/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2024-02-09T21:46:51.296 INFO:teuthology.orchestra.run.smithi006.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.5 MB/s | 300 kB 00:00 2024-02-09T21:46:51.346 INFO:teuthology.orchestra.run.smithi006.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-02-09T21:46:51.388 INFO:teuthology.orchestra.run.smithi006.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2024-02-09T21:46:51.430 INFO:teuthology.orchestra.run.smithi006.stdout:(4/6): libcephsqlite-17.2.7-505.g72ce7c1d.el9.x 484 kB/s | 162 kB 00:00 2024-02-09T21:46:51.455 INFO:teuthology.orchestra.run.smithi006.stdout:(5/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.2 MB/s | 217 kB 00:00 2024-02-09T21:46:53.885 INFO:teuthology.orchestra.run.smithi090.stdout:(6/6): ceph-test-17.2.7-505.g72ce7c1d.el9.x86_6 15 MB/s | 48 MB 00:03 2024-02-09T21:46:53.889 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:46:53.889 INFO:teuthology.orchestra.run.smithi090.stdout:Total 15 MB/s | 49 MB 00:03 2024-02-09T21:46:53.889 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-09T21:46:53.908 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-09T21:46:53.908 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-09T21:46:54.303 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-09T21:46:54.303 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-09T21:46:54.902 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-09T21:46:54.976 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-09T21:46:55.018 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-09T21:46:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-09T21:46:55.127 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-09T21:46:55.156 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephsqlite-2:17.2.7-505.g72ce7c1d.el9.x86_64 5/6 2024-02-09T21:46:56.314 INFO:teuthology.orchestra.run.smithi006.stdout:(6/6): ceph-test-17.2.7-505.g72ce7c1d.el9.x86_6 9.2 MB/s | 48 MB 00:05 2024-02-09T21:46:56.317 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:46:56.317 INFO:teuthology.orchestra.run.smithi006.stdout:Total 9.4 MB/s | 49 MB 00:05 2024-02-09T21:46:56.317 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-02-09T21:46:56.335 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-02-09T21:46:56.336 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-02-09T21:46:56.737 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-02-09T21:46:56.737 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-02-09T21:46:57.354 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-02-09T21:46:57.418 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-09T21:46:57.455 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-09T21:46:57.502 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-09T21:46:57.556 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-09T21:46:57.584 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephsqlite-2:17.2.7-505.g72ce7c1d.el9.x86_64 5/6 2024-02-09T21:46:59.652 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephsqlite-2:17.2.7-505.g72ce7c1d.el9.x86_64 5/6 2024-02-09T21:46:59.724 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-test-2:17.2.7-505.g72ce7c1d.el9.x86_64 6/6 2024-02-09T21:47:02.244 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephsqlite-2:17.2.7-505.g72ce7c1d.el9.x86_64 5/6 2024-02-09T21:47:02.328 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-test-2:17.2.7-505.g72ce7c1d.el9.x86_64 6/6 2024-02-09T21:47:02.614 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-test-2:17.2.7-505.g72ce7c1d.el9.x86_64 6/6 2024-02-09T21:47:02.614 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-test-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/6 2024-02-09T21:47:02.614 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephsqlite-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/6 2024-02-09T21:47:02.614 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-09T21:47:02.614 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-09T21:47:02.614 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-09T21:47:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-09T21:47:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:03.025 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-09T21:47:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:47:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: jq-1.6-16.el9.x86_64 2024-02-09T21:47:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:47:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-09T21:47:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-09T21:47:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-09T21:47:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:03.025 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-09T21:47:03.386 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph 2024-02-09T21:47:03.988 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:45 ago on Fri 09 Feb 2024 09:46:18 PM UTC. 2024-02-09T21:47:04.161 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-09T21:47:04.164 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2024-02-09T21:47:04.164 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-02-09T21:47:04.164 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2024-02-09T21:47:04.164 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-09T21:47:04.164 INFO:teuthology.orchestra.run.smithi090.stdout: ceph x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 6.5 k 2024-02-09T21:47:04.164 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-02-09T21:47:04.164 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 2.1 M 2024-02-09T21:47:04.164 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 1.4 M 2024-02-09T21:47:04.164 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 239 k 2024-02-09T21:47:04.165 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 3.7 M 2024-02-09T21:47:04.165 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 15 M 2024-02-09T21:47:04.165 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-09T21:47:04.165 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-09T21:47:04.165 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-09T21:47:04.165 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-09T21:47:04.165 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-09T21:47:04.165 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-09T21:47:04.165 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-09T21:47:04.165 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-09T21:47:04.165 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-09T21:47:04.165 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-09T21:47:04.165 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-09T21:47:04.165 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-09T21:47:04.166 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-09T21:47:04.166 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-09T21:47:04.166 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-09T21:47:04.166 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-09T21:47:04.166 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-09T21:47:04.166 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-09T21:47:04.166 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-09T21:47:04.166 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-09T21:47:04.166 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-09T21:47:04.166 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-09T21:47:04.166 INFO:teuthology.orchestra.run.smithi090.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-09T21:47:04.166 INFO:teuthology.orchestra.run.smithi090.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-09T21:47:04.166 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-09T21:47:04.166 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-09T21:47:04.167 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-09T21:47:04.167 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:04.167 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-09T21:47:04.167 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2024-02-09T21:47:04.167 INFO:teuthology.orchestra.run.smithi090.stdout:Install 33 Packages 2024-02-09T21:47:04.167 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:04.167 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 27 M 2024-02-09T21:47:04.168 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 99 M 2024-02-09T21:47:04.168 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-09T21:47:04.563 INFO:teuthology.orchestra.run.smithi090.stdout:(1/33): ceph-17.2.7-505.g72ce7c1d.el9.x86_64.rp 32 kB/s | 6.5 kB 00:00 2024-02-09T21:47:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-test-2:17.2.7-505.g72ce7c1d.el9.x86_64 6/6 2024-02-09T21:47:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-test-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/6 2024-02-09T21:47:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephsqlite-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/6 2024-02-09T21:47:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-09T21:47:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-09T21:47:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-09T21:47:05.047 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-09T21:47:05.048 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:05.048 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-02-09T21:47:05.048 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:47:05.048 INFO:teuthology.orchestra.run.smithi006.stdout: jq-1.6-16.el9.x86_64 2024-02-09T21:47:05.048 INFO:teuthology.orchestra.run.smithi006.stdout: libcephsqlite-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:47:05.048 INFO:teuthology.orchestra.run.smithi006.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-09T21:47:05.048 INFO:teuthology.orchestra.run.smithi006.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-09T21:47:05.048 INFO:teuthology.orchestra.run.smithi006.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-09T21:47:05.048 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:05.048 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-02-09T21:47:05.347 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph 2024-02-09T21:47:05.521 INFO:teuthology.orchestra.run.smithi090.stdout:(2/33): ceph-mds-17.2.7-505.g72ce7c1d.el9.x86_6 1.8 MB/s | 2.1 MB 00:01 2024-02-09T21:47:05.763 INFO:teuthology.orchestra.run.smithi090.stdout:(3/33): ceph-mgr-17.2.7-505.g72ce7c1d.el9.x86_6 1.0 MB/s | 1.4 MB 00:01 2024-02-09T21:47:05.830 INFO:teuthology.orchestra.run.smithi090.stdout:(4/33): ceph-mon-17.2.7-505.g72ce7c1d.el9.x86_6 2.9 MB/s | 3.7 MB 00:01 2024-02-09T21:47:05.952 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:00:43 ago on Fri 09 Feb 2024 09:46:22 PM UTC. 2024-02-09T21:47:05.964 INFO:teuthology.orchestra.run.smithi090.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 144 kB 00:00 2024-02-09T21:47:05.998 INFO:teuthology.orchestra.run.smithi090.stdout:(6/33): ceph-mgr-modules-core-17.2.7-505.g72ce7 1.0 MB/s | 239 kB 00:00 2024-02-09T21:47:06.023 INFO:teuthology.orchestra.run.smithi090.stdout:(7/33): python3-markupsafe-1.1.1-12.el9.x86_64. 537 kB/s | 32 kB 00:00 2024-02-09T21:47:06.048 INFO:teuthology.orchestra.run.smithi090.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 938 kB/s | 47 kB 00:00 2024-02-09T21:47:06.074 INFO:teuthology.orchestra.run.smithi090.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 769 kB/s | 39 kB 00:00 2024-02-09T21:47:06.107 INFO:teuthology.orchestra.run.smithi090.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-02-09T21:47:06.130 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-02-09T21:47:06.133 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================== 2024-02-09T21:47:06.133 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-02-09T21:47:06.133 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================== 2024-02-09T21:47:06.133 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-02-09T21:47:06.133 INFO:teuthology.orchestra.run.smithi006.stdout: ceph x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 6.5 k 2024-02-09T21:47:06.133 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-02-09T21:47:06.133 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 2.1 M 2024-02-09T21:47:06.134 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 1.4 M 2024-02-09T21:47:06.134 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 239 k 2024-02-09T21:47:06.134 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 3.7 M 2024-02-09T21:47:06.134 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 15 M 2024-02-09T21:47:06.134 INFO:teuthology.orchestra.run.smithi006.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-09T21:47:06.134 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-09T21:47:06.134 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-09T21:47:06.134 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-09T21:47:06.134 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-09T21:47:06.134 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-09T21:47:06.134 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-09T21:47:06.134 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-09T21:47:06.134 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-09T21:47:06.134 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-09T21:47:06.135 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-09T21:47:06.135 INFO:teuthology.orchestra.run.smithi006.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-09T21:47:06.135 INFO:teuthology.orchestra.run.smithi006.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-09T21:47:06.135 INFO:teuthology.orchestra.run.smithi006.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-09T21:47:06.135 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-09T21:47:06.135 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-09T21:47:06.135 INFO:teuthology.orchestra.run.smithi006.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-09T21:47:06.135 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-09T21:47:06.135 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-09T21:47:06.135 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-09T21:47:06.135 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-09T21:47:06.135 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-09T21:47:06.135 INFO:teuthology.orchestra.run.smithi006.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-09T21:47:06.135 INFO:teuthology.orchestra.run.smithi006.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-09T21:47:06.135 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-09T21:47:06.136 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-09T21:47:06.136 INFO:teuthology.orchestra.run.smithi006.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-09T21:47:06.136 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:06.136 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-02-09T21:47:06.136 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================== 2024-02-09T21:47:06.136 INFO:teuthology.orchestra.run.smithi006.stdout:Install 33 Packages 2024-02-09T21:47:06.136 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:06.136 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 27 M 2024-02-09T21:47:06.137 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 99 M 2024-02-09T21:47:06.137 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-02-09T21:47:06.175 INFO:teuthology.orchestra.run.smithi090.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-02-09T21:47:06.200 INFO:teuthology.orchestra.run.smithi090.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-02-09T21:47:06.225 INFO:teuthology.orchestra.run.smithi090.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-02-09T21:47:06.251 INFO:teuthology.orchestra.run.smithi090.stdout:(14/33): python3-requests-2.25.1-8.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2024-02-09T21:47:06.276 INFO:teuthology.orchestra.run.smithi090.stdout:(15/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-02-09T21:47:06.552 INFO:teuthology.orchestra.run.smithi006.stdout:(1/33): ceph-17.2.7-505.g72ce7c1d.el9.x86_64.rp 25 kB/s | 6.5 kB 00:00 2024-02-09T21:47:06.668 INFO:teuthology.orchestra.run.smithi090.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 104 kB/s | 43 kB 00:00 2024-02-09T21:47:06.836 INFO:teuthology.orchestra.run.smithi006.stdout:(2/33): ceph-mgr-17.2.7-505.g72ce7c1d.el9.x86_6 2.6 MB/s | 1.4 MB 00:00 2024-02-09T21:47:06.843 INFO:teuthology.orchestra.run.smithi090.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 303 kB/s | 172 kB 00:00 2024-02-09T21:47:06.927 INFO:teuthology.orchestra.run.smithi006.stdout:(3/33): ceph-mds-17.2.7-505.g72ce7c1d.el9.x86_6 3.3 MB/s | 2.1 MB 00:00 2024-02-09T21:47:06.944 INFO:teuthology.orchestra.run.smithi090.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-02-09T21:47:06.994 INFO:teuthology.orchestra.run.smithi006.stdout:(4/33): ceph-mgr-modules-core-17.2.7-505.g72ce7 3.5 MB/s | 239 kB 00:00 2024-02-09T21:47:07.028 INFO:teuthology.orchestra.run.smithi090.stdout:(19/33): python3-cherrypy-18.6.1-2.el9.noarch.r 998 kB/s | 358 kB 00:00 2024-02-09T21:47:07.053 INFO:teuthology.orchestra.run.smithi090.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 162 kB/s | 18 kB 00:00 2024-02-09T21:47:07.119 INFO:teuthology.orchestra.run.smithi006.stdout:(5/33): ceph-mon-17.2.7-505.g72ce7c1d.el9.x86_6 6.5 MB/s | 3.7 MB 00:00 2024-02-09T21:47:07.129 INFO:teuthology.orchestra.run.smithi090.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 228 kB/s | 23 kB 00:00 2024-02-09T21:47:07.145 INFO:teuthology.orchestra.run.smithi006.stdout:(6/33): python3-mako-1.1.4-6.el9.noarch.rpm 960 kB/s | 144 kB 00:00 2024-02-09T21:47:07.155 INFO:teuthology.orchestra.run.smithi090.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 191 kB/s | 19 kB 00:00 2024-02-09T21:47:07.170 INFO:teuthology.orchestra.run.smithi006.stdout:(7/33): python3-markupsafe-1.1.1-12.el9.x86_64. 629 kB/s | 32 kB 00:00 2024-02-09T21:47:07.195 INFO:teuthology.orchestra.run.smithi006.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 941 kB/s | 47 kB 00:00 2024-02-09T21:47:07.221 INFO:teuthology.orchestra.run.smithi006.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 769 kB/s | 39 kB 00:00 2024-02-09T21:47:07.230 INFO:teuthology.orchestra.run.smithi090.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2024-02-09T21:47:07.246 INFO:teuthology.orchestra.run.smithi006.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-02-09T21:47:07.264 INFO:teuthology.orchestra.run.smithi090.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 425 kB/s | 46 kB 00:00 2024-02-09T21:47:07.296 INFO:teuthology.orchestra.run.smithi006.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-02-09T21:47:07.322 INFO:teuthology.orchestra.run.smithi006.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-02-09T21:47:07.339 INFO:teuthology.orchestra.run.smithi090.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 726 kB/s | 79 kB 00:00 2024-02-09T21:47:07.347 INFO:teuthology.orchestra.run.smithi006.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-02-09T21:47:07.373 INFO:teuthology.orchestra.run.smithi006.stdout:(14/33): python3-requests-2.25.1-8.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2024-02-09T21:47:07.397 INFO:teuthology.orchestra.run.smithi006.stdout:(15/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-02-09T21:47:07.440 INFO:teuthology.orchestra.run.smithi090.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 164 kB/s | 16 kB 00:00 2024-02-09T21:47:07.473 INFO:teuthology.orchestra.run.smithi090.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-02-09T21:47:07.557 INFO:teuthology.orchestra.run.smithi090.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 769 kB/s | 90 kB 00:00 2024-02-09T21:47:07.582 INFO:teuthology.orchestra.run.smithi090.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 328 kB/s | 36 kB 00:00 2024-02-09T21:47:07.691 INFO:teuthology.orchestra.run.smithi090.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-02-09T21:47:07.758 INFO:teuthology.orchestra.run.smithi090.stdout:(31/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.4 MB/s | 427 kB 00:00 2024-02-09T21:47:07.765 INFO:teuthology.orchestra.run.smithi006.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 111 kB/s | 43 kB 00:00 2024-02-09T21:47:07.792 INFO:teuthology.orchestra.run.smithi090.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 199 kB/s | 20 kB 00:00 2024-02-09T21:47:08.015 INFO:teuthology.orchestra.run.smithi006.stdout:(17/33): ceph-osd-17.2.7-505.g72ce7c1d.el9.x86_ 13 MB/s | 15 MB 00:01 2024-02-09T21:47:08.040 INFO:teuthology.orchestra.run.smithi006.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 267 kB/s | 172 kB 00:00 2024-02-09T21:47:08.174 INFO:teuthology.orchestra.run.smithi006.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 132 kB/s | 18 kB 00:00 2024-02-09T21:47:08.216 INFO:teuthology.orchestra.run.smithi006.stdout:(20/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 53 kB/s | 11 kB 00:00 2024-02-09T21:47:08.241 INFO:teuthology.orchestra.run.smithi006.stdout:(21/33): python3-cherrypy-18.6.1-2.el9.noarch.r 753 kB/s | 358 kB 00:00 2024-02-09T21:47:08.299 INFO:teuthology.orchestra.run.smithi006.stdout:(22/33): python3-jaraco-collections-3.0.0-8.el9 185 kB/s | 23 kB 00:00 2024-02-09T21:47:08.375 INFO:teuthology.orchestra.run.smithi006.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 147 kB/s | 20 kB 00:00 2024-02-09T21:47:08.400 INFO:teuthology.orchestra.run.smithi006.stdout:(24/33): python3-jaraco-functools-3.5.0-2.el9.n 105 kB/s | 19 kB 00:00 2024-02-09T21:47:08.459 INFO:teuthology.orchestra.run.smithi006.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 291 kB/s | 46 kB 00:00 2024-02-09T21:47:08.526 INFO:teuthology.orchestra.run.smithi006.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 523 kB/s | 79 kB 00:00 2024-02-09T21:47:08.601 INFO:teuthology.orchestra.run.smithi006.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 115 kB/s | 16 kB 00:00 2024-02-09T21:47:08.685 INFO:teuthology.orchestra.run.smithi006.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 566 kB/s | 90 kB 00:00 2024-02-09T21:47:08.868 INFO:teuthology.orchestra.run.smithi006.stdout:(29/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2024-02-09T21:47:08.885 INFO:teuthology.orchestra.run.smithi006.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 126 kB/s | 36 kB 00:00 2024-02-09T21:47:08.935 INFO:teuthology.orchestra.run.smithi006.stdout:(31/33): python3-pecan-1.4.2-3.el9.noarch.rpm 509 kB/s | 272 kB 00:00 2024-02-09T21:47:09.002 INFO:teuthology.orchestra.run.smithi006.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 171 kB/s | 20 kB 00:00 2024-02-09T21:47:09.111 INFO:teuthology.orchestra.run.smithi006.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.7 MB/s | 427 kB 00:00 2024-02-09T21:47:09.114 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:47:09.114 INFO:teuthology.orchestra.run.smithi006.stdout:Total 9.0 MB/s | 27 MB 00:02 2024-02-09T21:47:09.115 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-02-09T21:47:09.157 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-02-09T21:47:09.157 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-02-09T21:47:09.512 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-02-09T21:47:09.512 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-02-09T21:47:10.576 INFO:teuthology.orchestra.run.smithi090.stdout:(33/33): ceph-osd-17.2.7-505.g72ce7c1d.el9.x86_ 3.0 MB/s | 15 MB 00:05 2024-02-09T21:47:10.582 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:47:10.582 INFO:teuthology.orchestra.run.smithi090.stdout:Total 4.2 MB/s | 27 MB 00:06 2024-02-09T21:47:10.582 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-09T21:47:10.620 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-09T21:47:10.620 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-09T21:47:10.742 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-02-09T21:47:10.797 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-02-09T21:47:10.838 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-02-09T21:47:10.879 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-02-09T21:47:10.967 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-09T21:47:10.967 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-09T21:47:10.975 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-02-09T21:47:11.061 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-02-09T21:47:11.108 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-02-09T21:47:11.210 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-02-09T21:47:11.267 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-02-09T21:47:11.304 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-02-09T21:47:11.355 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-02-09T21:47:11.429 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-02-09T21:47:11.627 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-02-09T21:47:11.678 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-02-09T21:47:11.716 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-02-09T21:47:11.751 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-02-09T21:47:11.781 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-02-09T21:47:11.860 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-02-09T21:47:11.899 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-09T21:47:11.933 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-02-09T21:47:11.973 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-02-09T21:47:12.008 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-02-09T21:47:12.042 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-02-09T21:47:12.135 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-02-09T21:47:12.173 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-02-09T21:47:12.244 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-09T21:47:12.248 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-02-09T21:47:12.312 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-02-09T21:47:12.353 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-02-09T21:47:12.389 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-02-09T21:47:12.394 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-02-09T21:47:12.485 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-02-09T21:47:12.519 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-09T21:47:12.556 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-02-09T21:47:12.598 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-02-09T21:47:12.678 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-02-09T21:47:12.679 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-505.g72ce7c1d.el9.n 28/33 2024-02-09T21:47:12.702 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-2:17.2.7-505.g72ce7c1d.el9.x86_64 29/33 2024-02-09T21:47:12.727 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-2:17.2.7-505.g72ce7c1d.el9.x86_64 29/33 2024-02-09T21:47:12.727 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T21:47:12.728 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-09T21:47:12.728 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-09T21:47:12.728 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-09T21:47:12.728 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:12.732 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-02-09T21:47:12.771 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-02-09T21:47:12.828 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-02-09T21:47:12.902 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-02-09T21:47:13.093 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-02-09T21:47:13.144 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-02-09T21:47:13.183 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-02-09T21:47:13.218 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-02-09T21:47:13.248 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-02-09T21:47:13.324 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-02-09T21:47:13.367 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-09T21:47:13.400 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-02-09T21:47:13.441 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-02-09T21:47:13.484 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-02-09T21:47:13.519 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-02-09T21:47:13.607 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-02-09T21:47:13.651 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-02-09T21:47:13.723 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-02-09T21:47:13.868 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-02-09T21:47:13.993 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-09T21:47:14.128 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-osd-2:17.2.7-505.g72ce7c1d.el9.x86_64 30/33 2024-02-09T21:47:14.143 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-505.g72ce7c1d.el9.n 28/33 2024-02-09T21:47:14.154 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-osd-2:17.2.7-505.g72ce7c1d.el9.x86_64 30/33 2024-02-09T21:47:14.154 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T21:47:14.155 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-09T21:47:14.155 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-09T21:47:14.155 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-09T21:47:14.155 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:14.174 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-2:17.2.7-505.g72ce7c1d.el9.x86_64 29/33 2024-02-09T21:47:14.199 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-2:17.2.7-505.g72ce7c1d.el9.x86_64 29/33 2024-02-09T21:47:14.199 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T21:47:14.200 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-09T21:47:14.200 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-09T21:47:14.200 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-09T21:47:14.200 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:14.546 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mon-2:17.2.7-505.g72ce7c1d.el9.x86_64 31/33 2024-02-09T21:47:14.568 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mon-2:17.2.7-505.g72ce7c1d.el9.x86_64 31/33 2024-02-09T21:47:14.568 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T21:47:14.568 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-09T21:47:14.569 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-09T21:47:14.569 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-09T21:47:14.569 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:14.771 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mds-2:17.2.7-505.g72ce7c1d.el9.x86_64 32/33 2024-02-09T21:47:14.794 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mds-2:17.2.7-505.g72ce7c1d.el9.x86_64 32/33 2024-02-09T21:47:14.794 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T21:47:14.794 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-09T21:47:14.794 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-09T21:47:14.794 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-09T21:47:14.794 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:14.854 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-2:17.2.7-505.g72ce7c1d.el9.x86_64 33/33 2024-02-09T21:47:15.593 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-osd-2:17.2.7-505.g72ce7c1d.el9.x86_64 30/33 2024-02-09T21:47:15.620 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-osd-2:17.2.7-505.g72ce7c1d.el9.x86_64 30/33 2024-02-09T21:47:15.620 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T21:47:15.621 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-09T21:47:15.621 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-09T21:47:15.621 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-09T21:47:15.621 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:15.986 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mon-2:17.2.7-505.g72ce7c1d.el9.x86_64 31/33 2024-02-09T21:47:16.009 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mon-2:17.2.7-505.g72ce7c1d.el9.x86_64 31/33 2024-02-09T21:47:16.009 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T21:47:16.009 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-09T21:47:16.010 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-09T21:47:16.010 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-09T21:47:16.010 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:16.220 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mds-2:17.2.7-505.g72ce7c1d.el9.x86_64 32/33 2024-02-09T21:47:16.243 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mds-2:17.2.7-505.g72ce7c1d.el9.x86_64 32/33 2024-02-09T21:47:16.243 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T21:47:16.243 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-09T21:47:16.243 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-09T21:47:16.243 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-09T21:47:16.243 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:16.309 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-2:17.2.7-505.g72ce7c1d.el9.x86_64 33/33 2024-02-09T21:47:16.556 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-2:17.2.7-505.g72ce7c1d.el9.x86_64 33/33 2024-02-09T21:47:16.556 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/33 2024-02-09T21:47:16.556 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mds-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/33 2024-02-09T21:47:16.556 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-2:17.2.7-505.g72ce7c1d.el9.x86_64 3/33 2024-02-09T21:47:16.556 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mon-2:17.2.7-505.g72ce7c1d.el9.x86_64 4/33 2024-02-09T21:47:16.556 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-osd-2:17.2.7-505.g72ce7c1d.el9.x86_64 5/33 2024-02-09T21:47:16.556 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-505.g72ce7c1d.el9.n 6/33 2024-02-09T21:47:16.556 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-02-09T21:47:16.556 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-02-09T21:47:16.557 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-02-09T21:47:16.557 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-02-09T21:47:16.557 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-02-09T21:47:16.557 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-02-09T21:47:16.557 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-02-09T21:47:16.557 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-02-09T21:47:16.557 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 15/33 2024-02-09T21:47:16.557 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/33 2024-02-09T21:47:16.557 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-02-09T21:47:16.557 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-09T21:47:16.557 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-02-09T21:47:16.557 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-02-09T21:47:16.557 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-02-09T21:47:16.557 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-02-09T21:47:16.557 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-02-09T21:47:16.558 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-02-09T21:47:16.558 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-02-09T21:47:16.558 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-02-09T21:47:16.558 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-09T21:47:16.558 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-02-09T21:47:16.558 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-02-09T21:47:16.558 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-02-09T21:47:16.558 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-02-09T21:47:16.558 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-02-09T21:47:17.665 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-02-09T21:47:17.665 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:17.665 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-02-09T21:47:17.665 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:47:17.666 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:47:17.666 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:47:17.666 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-09T21:47:17.666 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:47:17.666 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:47:17.666 INFO:teuthology.orchestra.run.smithi006.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-09T21:47:17.666 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-09T21:47:17.666 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-09T21:47:17.666 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-09T21:47:17.666 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-09T21:47:17.666 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-09T21:47:17.666 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-09T21:47:17.666 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-09T21:47:17.667 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-09T21:47:17.667 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-09T21:47:17.667 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-09T21:47:17.667 INFO:teuthology.orchestra.run.smithi006.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-09T21:47:17.667 INFO:teuthology.orchestra.run.smithi006.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-09T21:47:17.667 INFO:teuthology.orchestra.run.smithi006.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-09T21:47:17.667 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-09T21:47:17.667 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ply-3.11-14.el9.noarch 2024-02-09T21:47:17.667 INFO:teuthology.orchestra.run.smithi006.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-09T21:47:17.667 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-09T21:47:17.667 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-09T21:47:17.667 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-09T21:47:17.667 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-09T21:47:17.667 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-09T21:47:17.667 INFO:teuthology.orchestra.run.smithi006.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-09T21:47:17.668 INFO:teuthology.orchestra.run.smithi006.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-09T21:47:17.668 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-09T21:47:17.668 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-09T21:47:17.668 INFO:teuthology.orchestra.run.smithi006.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-09T21:47:17.668 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:17.668 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-02-09T21:47:18.121 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-base 2024-02-09T21:47:18.399 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-2:17.2.7-505.g72ce7c1d.el9.x86_64 33/33 2024-02-09T21:47:18.399 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/33 2024-02-09T21:47:18.399 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mds-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/33 2024-02-09T21:47:18.399 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-2:17.2.7-505.g72ce7c1d.el9.x86_64 3/33 2024-02-09T21:47:18.399 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mon-2:17.2.7-505.g72ce7c1d.el9.x86_64 4/33 2024-02-09T21:47:18.400 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-osd-2:17.2.7-505.g72ce7c1d.el9.x86_64 5/33 2024-02-09T21:47:18.400 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-505.g72ce7c1d.el9.n 6/33 2024-02-09T21:47:18.400 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-02-09T21:47:18.400 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-02-09T21:47:18.400 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-02-09T21:47:18.400 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-02-09T21:47:18.400 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-02-09T21:47:18.400 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-02-09T21:47:18.400 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-02-09T21:47:18.400 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-02-09T21:47:18.400 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 15/33 2024-02-09T21:47:18.400 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/33 2024-02-09T21:47:18.400 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-02-09T21:47:18.400 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-09T21:47:18.400 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-02-09T21:47:18.401 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-02-09T21:47:18.401 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-02-09T21:47:18.401 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-02-09T21:47:18.401 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-02-09T21:47:18.401 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-02-09T21:47:18.401 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-02-09T21:47:18.402 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-02-09T21:47:18.402 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-09T21:47:18.402 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-02-09T21:47:18.402 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-02-09T21:47:18.402 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-02-09T21:47:18.402 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-02-09T21:47:18.402 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-02-09T21:47:18.734 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:00:56 ago on Fri 09 Feb 2024 09:46:22 PM UTC. 2024-02-09T21:47:18.845 INFO:teuthology.orchestra.run.smithi006.stdout:Package ceph-base-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-09T21:47:18.899 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-02-09T21:47:18.900 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-02-09T21:47:18.900 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-02-09T21:47:18.960 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install cephadm 2024-02-09T21:47:19.349 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-02-09T21:47:19.349 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:19.349 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-09T21:47:19.350 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:47:19.350 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:47:19.350 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:47:19.350 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-09T21:47:19.350 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:47:19.350 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:47:19.350 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-09T21:47:19.350 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-09T21:47:19.350 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-09T21:47:19.350 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-09T21:47:19.350 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-09T21:47:19.350 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-09T21:47:19.350 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-09T21:47:19.350 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-09T21:47:19.350 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-09T21:47:19.351 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-09T21:47:19.351 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-09T21:47:19.351 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-09T21:47:19.351 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-09T21:47:19.351 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-09T21:47:19.351 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-09T21:47:19.351 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ply-3.11-14.el9.noarch 2024-02-09T21:47:19.351 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-09T21:47:19.351 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-09T21:47:19.351 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-09T21:47:19.351 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-09T21:47:19.351 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-09T21:47:19.351 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-09T21:47:19.351 INFO:teuthology.orchestra.run.smithi090.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-09T21:47:19.352 INFO:teuthology.orchestra.run.smithi090.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-09T21:47:19.352 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-09T21:47:19.352 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-09T21:47:19.352 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-09T21:47:19.352 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:19.352 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-09T21:47:19.550 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:00:57 ago on Fri 09 Feb 2024 09:46:22 PM UTC. 2024-02-09T21:47:19.715 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-02-09T21:47:19.715 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:47:19.715 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-02-09T21:47:19.716 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:47:19.716 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-02-09T21:47:19.716 INFO:teuthology.orchestra.run.smithi006.stdout: cephadm noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 80 k 2024-02-09T21:47:19.716 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:19.716 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-02-09T21:47:19.716 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:47:19.716 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-02-09T21:47:19.716 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:19.716 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 80 k 2024-02-09T21:47:19.716 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 364 k 2024-02-09T21:47:19.716 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-02-09T21:47:19.772 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-base 2024-02-09T21:47:20.028 INFO:teuthology.orchestra.run.smithi006.stdout:cephadm-17.2.7-505.g72ce7c1d.el9.noarch.rpm 258 kB/s | 80 kB 00:00 2024-02-09T21:47:20.029 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:47:20.029 INFO:teuthology.orchestra.run.smithi006.stdout:Total 257 kB/s | 80 kB 00:00 2024-02-09T21:47:20.029 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-02-09T21:47:20.035 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-02-09T21:47:20.035 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-02-09T21:47:20.037 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-02-09T21:47:20.037 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-02-09T21:47:20.109 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-02-09T21:47:20.358 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:02 ago on Fri 09 Feb 2024 09:46:18 PM UTC. 2024-02-09T21:47:20.465 INFO:teuthology.orchestra.run.smithi090.stdout:Package ceph-base-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-09T21:47:20.515 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-09T21:47:20.516 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-02-09T21:47:20.517 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-09T21:47:20.578 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 1/1 2024-02-09T21:47:20.617 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install cephadm 2024-02-09T21:47:20.677 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 1/1 2024-02-09T21:47:21.194 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:03 ago on Fri 09 Feb 2024 09:46:18 PM UTC. 2024-02-09T21:47:21.201 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 1/1 2024-02-09T21:47:21.353 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-09T21:47:21.354 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:47:21.354 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-02-09T21:47:21.354 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:47:21.354 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-09T21:47:21.354 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 80 k 2024-02-09T21:47:21.354 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:21.354 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-09T21:47:21.354 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:47:21.354 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-02-09T21:47:21.355 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:21.355 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 80 k 2024-02-09T21:47:21.355 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 364 k 2024-02-09T21:47:21.355 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-09T21:47:21.535 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 1/1 2024-02-09T21:47:21.536 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:21.536 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-02-09T21:47:21.536 INFO:teuthology.orchestra.run.smithi006.stdout: cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-09T21:47:21.536 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:21.536 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-02-09T21:47:21.658 INFO:teuthology.orchestra.run.smithi090.stdout:cephadm-17.2.7-505.g72ce7c1d.el9.noarch.rpm 266 kB/s | 80 kB 00:00 2024-02-09T21:47:21.658 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:47:21.659 INFO:teuthology.orchestra.run.smithi090.stdout:Total 264 kB/s | 80 kB 00:00 2024-02-09T21:47:21.659 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-09T21:47:21.664 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-09T21:47:21.664 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-09T21:47:21.666 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-09T21:47:21.666 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-09T21:47:21.716 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-immutable-object-cache 2024-02-09T21:47:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-09T21:47:22.232 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 1/1 2024-02-09T21:47:22.312 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:00 ago on Fri 09 Feb 2024 09:46:22 PM UTC. 2024-02-09T21:47:22.348 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 1/1 2024-02-09T21:47:22.474 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-02-09T21:47:22.475 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:47:22.475 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-02-09T21:47:22.475 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:47:22.475 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-02-09T21:47:22.475 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 137 k 2024-02-09T21:47:22.475 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:22.475 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-02-09T21:47:22.475 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:47:22.475 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-02-09T21:47:22.475 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:22.475 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 137 k 2024-02-09T21:47:22.475 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 379 k 2024-02-09T21:47:22.476 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-02-09T21:47:22.801 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-immutable-object-cache-17.2.7-505.g72ce7c1 421 kB/s | 137 kB 00:00 2024-02-09T21:47:22.802 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:47:22.802 INFO:teuthology.orchestra.run.smithi006.stdout:Total 419 kB/s | 137 kB 00:00 2024-02-09T21:47:22.802 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-02-09T21:47:22.810 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-02-09T21:47:22.810 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-02-09T21:47:22.830 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 1/1 2024-02-09T21:47:22.864 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-02-09T21:47:22.864 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-02-09T21:47:23.120 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-02-09T21:47:23.169 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-505.g72ce7c1d.e 1/1 2024-02-09T21:47:23.171 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 1/1 2024-02-09T21:47:23.171 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:23.172 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-09T21:47:23.172 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-09T21:47:23.172 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:23.172 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-09T21:47:23.194 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-505.g72ce7c1d.e 1/1 2024-02-09T21:47:23.195 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T21:47:23.195 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-09T21:47:23.195 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:23.462 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-immutable-object-cache 2024-02-09T21:47:24.059 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:06 ago on Fri 09 Feb 2024 09:46:18 PM UTC. 2024-02-09T21:47:24.220 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-09T21:47:24.221 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:47:24.221 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-02-09T21:47:24.221 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:47:24.221 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-09T21:47:24.221 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 137 k 2024-02-09T21:47:24.221 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:24.221 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-09T21:47:24.221 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:47:24.222 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-02-09T21:47:24.222 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:24.222 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 137 k 2024-02-09T21:47:24.222 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 379 k 2024-02-09T21:47:24.222 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-09T21:47:24.224 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-505.g72ce7c1d.e 1/1 2024-02-09T21:47:24.224 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:24.224 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-02-09T21:47:24.224 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:47:24.224 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:24.224 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-02-09T21:47:24.498 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr 2024-02-09T21:47:24.503 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-immutable-object-cache-17.2.7-505.g72ce7c1 488 kB/s | 137 kB 00:00 2024-02-09T21:47:24.503 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:47:24.503 INFO:teuthology.orchestra.run.smithi090.stdout:Total 485 kB/s | 137 kB 00:00 2024-02-09T21:47:24.503 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-09T21:47:24.511 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-09T21:47:24.511 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-09T21:47:24.563 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-09T21:47:24.563 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-09T21:47:24.701 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-09T21:47:24.770 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-505.g72ce7c1d.e 1/1 2024-02-09T21:47:24.794 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-505.g72ce7c1d.e 1/1 2024-02-09T21:47:24.795 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T21:47:24.795 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-09T21:47:24.795 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:25.088 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:03 ago on Fri 09 Feb 2024 09:46:22 PM UTC. 2024-02-09T21:47:25.198 INFO:teuthology.orchestra.run.smithi006.stdout:Package ceph-mgr-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-09T21:47:25.252 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-02-09T21:47:25.253 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-02-09T21:47:25.253 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-02-09T21:47:25.322 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-dashboard 2024-02-09T21:47:25.866 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-505.g72ce7c1d.e 1/1 2024-02-09T21:47:25.866 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:25.866 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-09T21:47:25.866 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:47:25.866 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:25.866 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-09T21:47:25.915 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:03 ago on Fri 09 Feb 2024 09:46:22 PM UTC. 2024-02-09T21:47:26.079 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-02-09T21:47:26.080 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:47:26.080 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-02-09T21:47:26.080 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:47:26.080 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-02-09T21:47:26.080 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 1.7 M 2024-02-09T21:47:26.080 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-02-09T21:47:26.080 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-grafana-dashboards noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 23 k 2024-02-09T21:47:26.080 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-prometheus-alerts noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 15 k 2024-02-09T21:47:26.080 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-02-09T21:47:26.080 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-09T21:47:26.081 INFO:teuthology.orchestra.run.smithi006.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-09T21:47:26.081 INFO:teuthology.orchestra.run.smithi006.stdout:Installing weak dependencies: 2024-02-09T21:47:26.081 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-02-09T21:47:26.081 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:26.081 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-02-09T21:47:26.081 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:47:26.081 INFO:teuthology.orchestra.run.smithi006.stdout:Install 7 Packages 2024-02-09T21:47:26.081 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:26.081 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 2.0 M 2024-02-09T21:47:26.082 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 8.1 M 2024-02-09T21:47:26.082 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-02-09T21:47:26.159 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr 2024-02-09T21:47:26.704 INFO:teuthology.orchestra.run.smithi006.stdout:(1/7): ceph-prometheus-alerts-17.2.7-505.g72ce7 61 kB/s | 15 kB 00:00 2024-02-09T21:47:26.746 INFO:teuthology.orchestra.run.smithi006.stdout:(2/7): ceph-grafana-dashboards-17.2.7-505.g72ce 83 kB/s | 23 kB 00:00 2024-02-09T21:47:26.748 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:08 ago on Fri 09 Feb 2024 09:46:18 PM UTC. 2024-02-09T21:47:26.821 INFO:teuthology.orchestra.run.smithi006.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 77 kB/s | 9.0 kB 00:00 2024-02-09T21:47:26.856 INFO:teuthology.orchestra.run.smithi090.stdout:Package ceph-mgr-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-09T21:47:26.888 INFO:teuthology.orchestra.run.smithi006.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 288 kB/s | 41 kB 00:00 2024-02-09T21:47:26.906 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-09T21:47:26.907 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-02-09T21:47:26.907 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-09T21:47:26.913 INFO:teuthology.orchestra.run.smithi006.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 335 kB/s | 31 kB 00:00 2024-02-09T21:47:26.969 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-dashboard 2024-02-09T21:47:27.089 INFO:teuthology.orchestra.run.smithi006.stdout:(6/7): ceph-mgr-dashboard-17.2.7-505.g72ce7c1d. 2.7 MB/s | 1.7 MB 00:00 2024-02-09T21:47:27.114 INFO:teuthology.orchestra.run.smithi006.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 837 kB/s | 188 kB 00:00 2024-02-09T21:47:27.115 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:47:27.116 INFO:teuthology.orchestra.run.smithi006.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2024-02-09T21:47:27.116 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-02-09T21:47:27.125 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-02-09T21:47:27.125 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-02-09T21:47:27.189 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-02-09T21:47:27.190 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-02-09T21:47:27.493 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-02-09T21:47:27.549 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:09 ago on Fri 09 Feb 2024 09:46:18 PM UTC. 2024-02-09T21:47:27.557 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-02-09T21:47:27.591 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-02-09T21:47:27.639 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-02-09T21:47:27.669 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-02-09T21:47:27.698 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-505.g72ce7c1d.el9.no 5/7 2024-02-09T21:47:27.708 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-09T21:47:27.709 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:47:27.709 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-02-09T21:47:27.709 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:47:27.709 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-09T21:47:27.709 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 1.7 M 2024-02-09T21:47:27.710 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-02-09T21:47:27.710 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 23 k 2024-02-09T21:47:27.710 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 15 k 2024-02-09T21:47:27.710 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-02-09T21:47:27.710 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-09T21:47:27.710 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-09T21:47:27.710 INFO:teuthology.orchestra.run.smithi090.stdout:Installing weak dependencies: 2024-02-09T21:47:27.710 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-02-09T21:47:27.710 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:27.710 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-09T21:47:27.710 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:47:27.710 INFO:teuthology.orchestra.run.smithi090.stdout:Install 7 Packages 2024-02-09T21:47:27.710 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:27.711 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 2.0 M 2024-02-09T21:47:27.711 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 8.1 M 2024-02-09T21:47:27.711 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-09T21:47:27.940 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-505.g72ce7c1d.el9.n 6/7 2024-02-09T21:47:27.967 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-505.g72ce7c1d.el9.noarch 7/7 2024-02-09T21:47:27.987 INFO:teuthology.orchestra.run.smithi090.stdout:(1/7): ceph-prometheus-alerts-17.2.7-505.g72ce7 63 kB/s | 15 kB 00:00 2024-02-09T21:47:28.012 INFO:teuthology.orchestra.run.smithi090.stdout:(2/7): ceph-grafana-dashboards-17.2.7-505.g72ce 91 kB/s | 23 kB 00:00 2024-02-09T21:47:28.070 INFO:teuthology.orchestra.run.smithi090.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 108 kB/s | 9.0 kB 00:00 2024-02-09T21:47:28.104 INFO:teuthology.orchestra.run.smithi090.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 446 kB/s | 41 kB 00:00 2024-02-09T21:47:28.129 INFO:teuthology.orchestra.run.smithi090.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 523 kB/s | 31 kB 00:00 2024-02-09T21:47:28.188 INFO:teuthology.orchestra.run.smithi090.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-02-09T21:47:28.414 INFO:teuthology.orchestra.run.smithi090.stdout:(7/7): ceph-mgr-dashboard-17.2.7-505.g72ce7c1d. 2.6 MB/s | 1.7 MB 00:00 2024-02-09T21:47:28.415 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:47:28.416 INFO:teuthology.orchestra.run.smithi090.stdout:Total 2.8 MB/s | 2.0 MB 00:00 2024-02-09T21:47:28.416 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-09T21:47:28.424 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-09T21:47:28.424 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-09T21:47:28.486 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-505.g72ce7c1d.el9.noarch 7/7 2024-02-09T21:47:28.486 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-505.g72ce7c1d.el9.n 1/7 2024-02-09T21:47:28.486 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-505.g72ce7c1d.el9.noarch 2/7 2024-02-09T21:47:28.486 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-505.g72ce7c1d.el9.no 3/7 2024-02-09T21:47:28.487 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-02-09T21:47:28.487 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-02-09T21:47:28.487 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-02-09T21:47:28.496 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-09T21:47:28.496 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-09T21:47:28.798 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-09T21:47:28.810 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-02-09T21:47:28.810 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:28.810 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-02-09T21:47:28.811 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-grafana-dashboards-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-09T21:47:28.811 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-09T21:47:28.811 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-prometheus-alerts-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-09T21:47:28.811 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-02-09T21:47:28.811 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-02-09T21:47:28.811 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-09T21:47:28.811 INFO:teuthology.orchestra.run.smithi006.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-09T21:47:28.811 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:28.811 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-02-09T21:47:28.865 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-02-09T21:47:28.899 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-02-09T21:47:28.946 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-02-09T21:47:28.994 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-02-09T21:47:29.031 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-09T21:47:29.032 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-505.g72ce7c1d.el9.no 5/7 2024-02-09T21:47:29.261 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-505.g72ce7c1d.el9.n 6/7 2024-02-09T21:47:29.293 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-505.g72ce7c1d.el9.noarch 7/7 2024-02-09T21:47:29.621 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:07 ago on Fri 09 Feb 2024 09:46:22 PM UTC. 2024-02-09T21:47:29.789 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-02-09T21:47:29.790 INFO:teuthology.orchestra.run.smithi006.stdout:=========================================================================================== 2024-02-09T21:47:29.790 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-02-09T21:47:29.790 INFO:teuthology.orchestra.run.smithi006.stdout:=========================================================================================== 2024-02-09T21:47:29.790 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-02-09T21:47:29.790 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 7.4 M 2024-02-09T21:47:29.790 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-02-09T21:47:29.790 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-09T21:47:29.790 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-09T21:47:29.790 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-09T21:47:29.790 INFO:teuthology.orchestra.run.smithi006.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-09T21:47:29.791 INFO:teuthology.orchestra.run.smithi006.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-09T21:47:29.791 INFO:teuthology.orchestra.run.smithi006.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-09T21:47:29.791 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-09T21:47:29.791 INFO:teuthology.orchestra.run.smithi006.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-09T21:47:29.791 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-09T21:47:29.791 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-09T21:47:29.791 INFO:teuthology.orchestra.run.smithi006.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-09T21:47:29.791 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:29.791 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-02-09T21:47:29.791 INFO:teuthology.orchestra.run.smithi006.stdout:=========================================================================================== 2024-02-09T21:47:29.791 INFO:teuthology.orchestra.run.smithi006.stdout:Install 12 Packages 2024-02-09T21:47:29.791 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:29.792 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 37 M 2024-02-09T21:47:29.792 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 209 M 2024-02-09T21:47:29.792 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-02-09T21:47:29.976 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-505.g72ce7c1d.el9.noarch 7/7 2024-02-09T21:47:29.976 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-505.g72ce7c1d.el9.n 1/7 2024-02-09T21:47:29.976 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-505.g72ce7c1d.el9.noarch 2/7 2024-02-09T21:47:29.976 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-505.g72ce7c1d.el9.no 3/7 2024-02-09T21:47:29.976 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-02-09T21:47:29.976 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-02-09T21:47:29.976 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-02-09T21:47:30.156 INFO:teuthology.orchestra.run.smithi006.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 173 kB/s | 32 kB 00:00 2024-02-09T21:47:30.204 INFO:teuthology.orchestra.run.smithi006.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 343 kB/s | 17 kB 00:00 2024-02-09T21:47:30.271 INFO:teuthology.orchestra.run.smithi006.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 513 kB/s | 34 kB 00:00 2024-02-09T21:47:30.306 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-02-09T21:47:30.306 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:30.306 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-09T21:47:30.306 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-09T21:47:30.306 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-09T21:47:30.307 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-09T21:47:30.307 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-02-09T21:47:30.307 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-02-09T21:47:30.307 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-09T21:47:30.307 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-09T21:47:30.307 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:30.307 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-09T21:47:30.372 INFO:teuthology.orchestra.run.smithi006.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.5 MB/s | 3.0 MB 00:00 2024-02-09T21:47:30.542 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-09T21:47:30.630 INFO:teuthology.orchestra.run.smithi006.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 13 MB/s | 4.7 MB 00:00 2024-02-09T21:47:30.689 INFO:teuthology.orchestra.run.smithi006.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 15 MB/s | 4.8 MB 00:00 2024-02-09T21:47:30.739 INFO:teuthology.orchestra.run.smithi006.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.0 MB/s | 217 kB 00:00 2024-02-09T21:47:31.022 INFO:teuthology.orchestra.run.smithi006.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.8 MB/s | 802 kB 00:00 2024-02-09T21:47:31.114 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:13 ago on Fri 09 Feb 2024 09:46:18 PM UTC. 2024-02-09T21:47:31.192 INFO:teuthology.orchestra.run.smithi006.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-02-09T21:47:31.215 INFO:teuthology.orchestra.run.smithi006.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 194 kB 00:00 2024-02-09T21:47:31.275 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-09T21:47:31.276 INFO:teuthology.orchestra.run.smithi090.stdout:=========================================================================================== 2024-02-09T21:47:31.277 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-02-09T21:47:31.277 INFO:teuthology.orchestra.run.smithi090.stdout:=========================================================================================== 2024-02-09T21:47:31.277 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-09T21:47:31.277 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 7.4 M 2024-02-09T21:47:31.277 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-02-09T21:47:31.277 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-09T21:47:31.277 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-09T21:47:31.277 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-09T21:47:31.277 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-09T21:47:31.277 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-09T21:47:31.277 INFO:teuthology.orchestra.run.smithi090.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-09T21:47:31.277 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-09T21:47:31.277 INFO:teuthology.orchestra.run.smithi090.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-09T21:47:31.277 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-09T21:47:31.277 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-09T21:47:31.278 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-09T21:47:31.278 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:31.278 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-09T21:47:31.278 INFO:teuthology.orchestra.run.smithi090.stdout:=========================================================================================== 2024-02-09T21:47:31.278 INFO:teuthology.orchestra.run.smithi090.stdout:Install 12 Packages 2024-02-09T21:47:31.278 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:31.278 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 37 M 2024-02-09T21:47:31.279 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 209 M 2024-02-09T21:47:31.279 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-09T21:47:31.582 INFO:teuthology.orchestra.run.smithi090.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 146 kB/s | 32 kB 00:00 2024-02-09T21:47:31.615 INFO:teuthology.orchestra.run.smithi006.stdout:(11/12): ceph-mgr-diskprediction-local-17.2.7-5 4.5 MB/s | 7.4 MB 00:01 2024-02-09T21:47:31.615 INFO:teuthology.orchestra.run.smithi090.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 524 kB/s | 17 kB 00:00 2024-02-09T21:47:31.649 INFO:teuthology.orchestra.run.smithi090.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-02-09T21:47:31.858 INFO:teuthology.orchestra.run.smithi090.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2024-02-09T21:47:31.917 INFO:teuthology.orchestra.run.smithi090.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.4 MB/s | 3.0 MB 00:00 2024-02-09T21:47:32.051 INFO:teuthology.orchestra.run.smithi090.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 4.8 MB 00:00 2024-02-09T21:47:32.084 INFO:teuthology.orchestra.run.smithi090.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.3 MB/s | 217 kB 00:00 2024-02-09T21:47:32.235 INFO:teuthology.orchestra.run.smithi090.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 5.3 MB/s | 802 kB 00:00 2024-02-09T21:47:32.443 INFO:teuthology.orchestra.run.smithi090.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 40 MB/s | 16 MB 00:00 2024-02-09T21:47:32.477 INFO:teuthology.orchestra.run.smithi090.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 809 kB/s | 194 kB 00:00 2024-02-09T21:47:32.694 INFO:teuthology.orchestra.run.smithi090.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 982 kB/s | 245 kB 00:00 2024-02-09T21:47:32.877 INFO:teuthology.orchestra.run.smithi090.stdout:(12/12): ceph-mgr-diskprediction-local-17.2.7-5 4.9 MB/s | 7.4 MB 00:01 2024-02-09T21:47:32.880 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:47:32.880 INFO:teuthology.orchestra.run.smithi090.stdout:Total 23 MB/s | 37 MB 00:01 2024-02-09T21:47:32.899 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-09T21:47:32.923 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-09T21:47:32.923 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-09T21:47:33.057 INFO:teuthology.orchestra.run.smithi006.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 132 kB/s | 245 kB 00:01 2024-02-09T21:47:33.060 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:47:33.060 INFO:teuthology.orchestra.run.smithi006.stdout:Total 11 MB/s | 37 MB 00:03 2024-02-09T21:47:33.083 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-02-09T21:47:33.111 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-02-09T21:47:33.111 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-02-09T21:47:33.352 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-09T21:47:33.352 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-09T21:47:33.551 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-02-09T21:47:33.552 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-02-09T21:47:34.044 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-09T21:47:34.136 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-09T21:47:34.167 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-09T21:47:34.260 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-09T21:47:34.271 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-02-09T21:47:34.304 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-09T21:47:34.352 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-09T21:47:34.395 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-09T21:47:34.479 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-09T21:47:34.492 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-09T21:47:34.511 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-09T21:47:34.540 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-09T21:47:34.596 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-09T21:47:34.723 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-09T21:47:34.755 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-09T21:47:34.834 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-09T21:47:35.263 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-09T21:47:35.376 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-09T21:47:35.531 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-09T21:47:35.648 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-09T21:47:36.665 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-09T21:47:36.965 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-09T21:47:37.688 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-09T21:47:37.727 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-505.g72ce7c 12/12 2024-02-09T21:47:38.042 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-09T21:47:38.071 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-505.g72ce7c 12/12 2024-02-09T21:47:40.564 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-505.g72ce7c 12/12 2024-02-09T21:47:40.564 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-505.g72ce7c 1/12 2024-02-09T21:47:40.564 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-09T21:47:40.565 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-09T21:47:40.565 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-09T21:47:40.565 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-09T21:47:40.565 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-09T21:47:40.565 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-09T21:47:40.565 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-09T21:47:40.565 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-09T21:47:40.565 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-09T21:47:40.565 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-09T21:47:41.044 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-09T21:47:41.044 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:41.044 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-02-09T21:47:41.044 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-diskprediction-local-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-09T21:47:41.044 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-09T21:47:41.044 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-09T21:47:41.044 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-09T21:47:41.044 INFO:teuthology.orchestra.run.smithi006.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-09T21:47:41.044 INFO:teuthology.orchestra.run.smithi006.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-09T21:47:41.044 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-09T21:47:41.044 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-09T21:47:41.045 INFO:teuthology.orchestra.run.smithi006.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-09T21:47:41.045 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-09T21:47:41.045 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-09T21:47:41.045 INFO:teuthology.orchestra.run.smithi006.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-09T21:47:41.045 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:41.045 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-02-09T21:47:41.143 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-505.g72ce7c 12/12 2024-02-09T21:47:41.143 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-505.g72ce7c 1/12 2024-02-09T21:47:41.143 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-09T21:47:41.143 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-09T21:47:41.143 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-09T21:47:41.144 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-09T21:47:41.144 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-09T21:47:41.144 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-09T21:47:41.144 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-09T21:47:41.144 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-09T21:47:41.144 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-09T21:47:41.144 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-09T21:47:41.397 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-rook 2024-02-09T21:47:41.616 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-09T21:47:41.617 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:41.617 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-09T21:47:41.617 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-09T21:47:41.617 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-09T21:47:41.617 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-09T21:47:41.617 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-09T21:47:41.617 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-09T21:47:41.617 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-09T21:47:41.617 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-09T21:47:41.617 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-09T21:47:41.617 INFO:teuthology.orchestra.run.smithi090.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-09T21:47:41.617 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-09T21:47:41.618 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-09T21:47:41.618 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-09T21:47:41.618 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:41.618 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-09T21:47:41.993 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:19 ago on Fri 09 Feb 2024 09:46:22 PM UTC. 2024-02-09T21:47:42.022 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-rook 2024-02-09T21:47:42.158 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-02-09T21:47:42.160 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================= 2024-02-09T21:47:42.160 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-02-09T21:47:42.160 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================= 2024-02-09T21:47:42.160 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-02-09T21:47:42.160 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-rook noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 47 k 2024-02-09T21:47:42.160 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-02-09T21:47:42.160 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-09T21:47:42.160 INFO:teuthology.orchestra.run.smithi006.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-09T21:47:42.160 INFO:teuthology.orchestra.run.smithi006.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-09T21:47:42.160 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-09T21:47:42.160 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-09T21:47:42.160 INFO:teuthology.orchestra.run.smithi006.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-09T21:47:42.161 INFO:teuthology.orchestra.run.smithi006.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-09T21:47:42.161 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-09T21:47:42.161 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-09T21:47:42.161 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-09T21:47:42.161 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-09T21:47:42.161 INFO:teuthology.orchestra.run.smithi006.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-09T21:47:42.161 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:42.161 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-02-09T21:47:42.161 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================= 2024-02-09T21:47:42.161 INFO:teuthology.orchestra.run.smithi006.stdout:Install 13 Packages 2024-02-09T21:47:42.161 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:42.162 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 2.0 M 2024-02-09T21:47:42.162 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 26 M 2024-02-09T21:47:42.162 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-02-09T21:47:42.446 INFO:teuthology.orchestra.run.smithi006.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 131 kB/s | 18 kB 00:00 2024-02-09T21:47:42.471 INFO:teuthology.orchestra.run.smithi006.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 149 kB/s | 24 kB 00:00 2024-02-09T21:47:42.522 INFO:teuthology.orchestra.run.smithi006.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-02-09T21:47:42.555 INFO:teuthology.orchestra.run.smithi006.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-02-09T21:47:42.581 INFO:teuthology.orchestra.run.smithi006.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-02-09T21:47:42.598 INFO:teuthology.orchestra.run.smithi006.stdout:(6/13): ceph-mgr-rook-17.2.7-505.g72ce7c1d.el9. 164 kB/s | 47 kB 00:00 2024-02-09T21:47:42.607 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:24 ago on Fri 09 Feb 2024 09:46:18 PM UTC. 2024-02-09T21:47:42.614 INFO:teuthology.orchestra.run.smithi006.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 733 kB/s | 43 kB 00:00 2024-02-09T21:47:42.765 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-09T21:47:42.766 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================= 2024-02-09T21:47:42.766 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-02-09T21:47:42.766 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================= 2024-02-09T21:47:42.766 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-09T21:47:42.766 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 47 k 2024-02-09T21:47:42.766 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-02-09T21:47:42.766 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-09T21:47:42.766 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-09T21:47:42.767 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-09T21:47:42.767 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-09T21:47:42.767 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-09T21:47:42.767 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-09T21:47:42.767 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-09T21:47:42.767 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-09T21:47:42.767 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-09T21:47:42.767 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-09T21:47:42.767 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-09T21:47:42.767 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-09T21:47:42.767 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:42.767 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-09T21:47:42.767 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================= 2024-02-09T21:47:42.767 INFO:teuthology.orchestra.run.smithi090.stdout:Install 13 Packages 2024-02-09T21:47:42.767 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:42.768 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 2.0 M 2024-02-09T21:47:42.768 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 26 M 2024-02-09T21:47:42.768 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-09T21:47:42.798 INFO:teuthology.orchestra.run.smithi006.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 82 kB/s | 15 kB 00:00 2024-02-09T21:47:42.823 INFO:teuthology.orchestra.run.smithi006.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 133 kB/s | 32 kB 00:00 2024-02-09T21:47:42.882 INFO:teuthology.orchestra.run.smithi006.stdout:(10/13): python3-google-auth-2.27.0-1.el9.noarc 759 kB/s | 203 kB 00:00 2024-02-09T21:47:42.957 INFO:teuthology.orchestra.run.smithi006.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 6.4 MB/s | 1.0 MB 00:00 2024-02-09T21:47:42.982 INFO:teuthology.orchestra.run.smithi006.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 372 kB/s | 59 kB 00:00 2024-02-09T21:47:43.007 INFO:teuthology.orchestra.run.smithi006.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 718 kB/s | 90 kB 00:00 2024-02-09T21:47:43.012 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:47:43.012 INFO:teuthology.orchestra.run.smithi006.stdout:Total 2.4 MB/s | 2.0 MB 00:00 2024-02-09T21:47:43.012 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-02-09T21:47:43.026 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-02-09T21:47:43.026 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-02-09T21:47:43.127 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-02-09T21:47:43.128 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-02-09T21:47:43.301 INFO:teuthology.orchestra.run.smithi090.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 110 kB/s | 18 kB 00:00 2024-02-09T21:47:43.326 INFO:teuthology.orchestra.run.smithi090.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 128 kB/s | 24 kB 00:00 2024-02-09T21:47:43.386 INFO:teuthology.orchestra.run.smithi090.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 174 kB 00:00 2024-02-09T21:47:43.412 INFO:teuthology.orchestra.run.smithi090.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-02-09T21:47:43.437 INFO:teuthology.orchestra.run.smithi090.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-02-09T21:47:43.462 INFO:teuthology.orchestra.run.smithi090.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-02-09T21:47:43.488 INFO:teuthology.orchestra.run.smithi090.stdout:(7/13): ceph-mgr-rook-17.2.7-505.g72ce7c1d.el9. 135 kB/s | 47 kB 00:00 2024-02-09T21:47:43.546 INFO:teuthology.orchestra.run.smithi090.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 179 kB/s | 15 kB 00:00 2024-02-09T21:47:43.572 INFO:teuthology.orchestra.run.smithi090.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 239 kB/s | 32 kB 00:00 2024-02-09T21:47:43.639 INFO:teuthology.orchestra.run.smithi090.stdout:(10/13): python3-google-auth-2.27.0-1.el9.noarc 1.3 MB/s | 203 kB 00:00 2024-02-09T21:47:43.662 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-02-09T21:47:43.664 INFO:teuthology.orchestra.run.smithi090.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 642 kB/s | 59 kB 00:00 2024-02-09T21:47:43.689 INFO:teuthology.orchestra.run.smithi090.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-02-09T21:47:43.765 INFO:teuthology.orchestra.run.smithi090.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 4.7 MB/s | 1.0 MB 00:00 2024-02-09T21:47:43.766 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:47:43.767 INFO:teuthology.orchestra.run.smithi090.stdout:Total 2.0 MB/s | 2.0 MB 00:00 2024-02-09T21:47:43.767 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-09T21:47:43.781 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-09T21:47:43.781 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-09T21:47:43.852 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-09T21:47:43.879 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-09T21:47:43.879 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-09T21:47:43.912 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-09T21:47:43.967 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-09T21:47:44.002 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-09T21:47:44.045 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-09T21:47:44.144 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-09T21:47:44.296 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-09T21:47:44.363 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-09T21:47:44.400 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-09T21:47:44.587 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-09T21:47:44.650 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-09T21:47:44.706 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-09T21:47:44.743 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-09T21:47:44.785 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-09T21:47:44.878 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-09T21:47:44.988 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-09T21:47:45.045 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-09T21:47:45.160 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-09T21:47:45.209 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-09T21:47:45.242 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-09T21:47:45.289 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-09T21:47:45.311 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-rook-2:17.2.7-505.g72ce7c1d.el9.noarch 13/13 2024-02-09T21:47:45.772 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-09T21:47:45.854 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-09T21:47:45.887 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-09T21:47:45.933 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-09T21:47:45.966 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-rook-2:17.2.7-505.g72ce7c1d.el9.noarch 13/13 2024-02-09T21:47:46.056 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-505.g72ce7c1d.el9.noarch 13/13 2024-02-09T21:47:46.056 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-rook-2:17.2.7-505.g72ce7c1d.el9.noarch 1/13 2024-02-09T21:47:46.056 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-09T21:47:46.056 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-09T21:47:46.056 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-09T21:47:46.057 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-09T21:47:46.057 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-09T21:47:46.057 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-09T21:47:46.057 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-09T21:47:46.057 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-09T21:47:46.057 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-09T21:47:46.057 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-09T21:47:46.057 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-09T21:47:46.434 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-09T21:47:46.434 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:46.434 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-02-09T21:47:46.434 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-rook-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-09T21:47:46.434 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-09T21:47:46.434 INFO:teuthology.orchestra.run.smithi006.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-09T21:47:46.434 INFO:teuthology.orchestra.run.smithi006.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-09T21:47:46.434 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-09T21:47:46.434 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-09T21:47:46.434 INFO:teuthology.orchestra.run.smithi006.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-09T21:47:46.434 INFO:teuthology.orchestra.run.smithi006.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-09T21:47:46.435 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-09T21:47:46.435 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-09T21:47:46.435 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-09T21:47:46.435 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-09T21:47:46.435 INFO:teuthology.orchestra.run.smithi006.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-09T21:47:46.435 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:46.435 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-02-09T21:47:46.683 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-cephadm 2024-02-09T21:47:46.989 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-505.g72ce7c1d.el9.noarch 13/13 2024-02-09T21:47:46.989 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-rook-2:17.2.7-505.g72ce7c1d.el9.noarch 1/13 2024-02-09T21:47:46.989 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-09T21:47:46.989 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-09T21:47:46.989 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-09T21:47:46.989 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-09T21:47:46.989 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-09T21:47:46.990 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-09T21:47:46.990 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-09T21:47:46.990 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-09T21:47:46.990 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-09T21:47:46.990 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-09T21:47:46.990 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-09T21:47:47.282 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:25 ago on Fri 09 Feb 2024 09:46:22 PM UTC. 2024-02-09T21:47:47.357 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-09T21:47:47.357 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:47.357 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-09T21:47:47.357 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-09T21:47:47.357 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-09T21:47:47.357 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-09T21:47:47.357 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-09T21:47:47.357 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-09T21:47:47.357 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-09T21:47:47.357 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-09T21:47:47.357 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-09T21:47:47.357 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-09T21:47:47.357 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-09T21:47:47.358 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-09T21:47:47.358 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-09T21:47:47.358 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-09T21:47:47.358 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:47.358 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-09T21:47:47.447 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-02-09T21:47:47.448 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================= 2024-02-09T21:47:47.448 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-02-09T21:47:47.448 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================= 2024-02-09T21:47:47.448 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-02-09T21:47:47.448 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-cephadm noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 121 k 2024-02-09T21:47:47.448 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-02-09T21:47:47.448 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-02-09T21:47:47.448 INFO:teuthology.orchestra.run.smithi006.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-09T21:47:47.448 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-09T21:47:47.448 INFO:teuthology.orchestra.run.smithi006.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-09T21:47:47.448 INFO:teuthology.orchestra.run.smithi006.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-09T21:47:47.448 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:47.448 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-02-09T21:47:47.449 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================= 2024-02-09T21:47:47.449 INFO:teuthology.orchestra.run.smithi006.stdout:Install 6 Packages 2024-02-09T21:47:47.449 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:47.449 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 6.8 M 2024-02-09T21:47:47.449 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 33 M 2024-02-09T21:47:47.449 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-02-09T21:47:47.648 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-cephadm 2024-02-09T21:47:47.767 INFO:teuthology.orchestra.run.smithi006.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 227 kB 00:00 2024-02-09T21:47:47.909 INFO:teuthology.orchestra.run.smithi006.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-02-09T21:47:47.959 INFO:teuthology.orchestra.run.smithi006.stdout:(3/6): ceph-mgr-cephadm-17.2.7-505.g72ce7c1d.el 336 kB/s | 121 kB 00:00 2024-02-09T21:47:48.001 INFO:teuthology.orchestra.run.smithi006.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 628 kB/s | 58 kB 00:00 2024-02-09T21:47:48.026 INFO:teuthology.orchestra.run.smithi006.stdout:(5/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 2.1 MB/s | 548 kB 00:00 2024-02-09T21:47:48.068 INFO:teuthology.orchestra.run.smithi006.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 469 kB/s | 51 kB 00:00 2024-02-09T21:47:48.069 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:47:48.070 INFO:teuthology.orchestra.run.smithi006.stdout:Total 11 MB/s | 6.8 MB 00:00 2024-02-09T21:47:48.070 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-02-09T21:47:48.079 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-02-09T21:47:48.079 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-02-09T21:47:48.172 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-02-09T21:47:48.173 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-02-09T21:47:48.233 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:30 ago on Fri 09 Feb 2024 09:46:18 PM UTC. 2024-02-09T21:47:48.389 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-09T21:47:48.390 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================= 2024-02-09T21:47:48.390 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-02-09T21:47:48.390 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================= 2024-02-09T21:47:48.390 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-09T21:47:48.390 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm noarch 2:17.2.7-505.g72ce7c1d.el9 ceph-noarch 121 k 2024-02-09T21:47:48.390 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-02-09T21:47:48.390 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-02-09T21:47:48.390 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-09T21:47:48.391 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-09T21:47:48.391 INFO:teuthology.orchestra.run.smithi090.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-09T21:47:48.391 INFO:teuthology.orchestra.run.smithi090.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-09T21:47:48.391 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:48.391 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-09T21:47:48.391 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================= 2024-02-09T21:47:48.391 INFO:teuthology.orchestra.run.smithi090.stdout:Install 6 Packages 2024-02-09T21:47:48.391 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:48.392 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 6.8 M 2024-02-09T21:47:48.392 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 33 M 2024-02-09T21:47:48.392 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-09T21:47:48.478 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-02-09T21:47:48.646 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-09T21:47:48.649 INFO:teuthology.orchestra.run.smithi090.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.2 MB/s | 227 kB 00:00 2024-02-09T21:47:48.695 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-02-09T21:47:48.799 INFO:teuthology.orchestra.run.smithi090.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2024-02-09T21:47:48.825 INFO:teuthology.orchestra.run.smithi090.stdout:(3/6): ceph-mgr-cephadm-17.2.7-505.g72ce7c1d.el 338 kB/s | 121 kB 00:00 2024-02-09T21:47:48.959 INFO:teuthology.orchestra.run.smithi090.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 363 kB/s | 58 kB 00:00 2024-02-09T21:47:48.976 INFO:teuthology.orchestra.run.smithi090.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 339 kB/s | 51 kB 00:00 2024-02-09T21:47:49.009 INFO:teuthology.orchestra.run.smithi090.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.5 MB/s | 548 kB 00:00 2024-02-09T21:47:49.011 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:47:49.011 INFO:teuthology.orchestra.run.smithi090.stdout:Total 11 MB/s | 6.8 MB 00:00 2024-02-09T21:47:49.011 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-09T21:47:49.020 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-09T21:47:49.020 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-09T21:47:49.109 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-09T21:47:49.109 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-09T21:47:49.251 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-09T21:47:49.333 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-09T21:47:49.387 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-02-09T21:47:49.398 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-09T21:47:49.416 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 6/6 2024-02-09T21:47:49.574 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-09T21:47:49.628 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-02-09T21:47:50.153 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-09T21:47:50.202 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 6/6 2024-02-09T21:47:50.202 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 1/6 2024-02-09T21:47:50.202 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-09T21:47:50.202 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-02-09T21:47:50.202 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-02-09T21:47:50.202 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-09T21:47:50.247 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-09T21:47:50.312 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-02-09T21:47:50.342 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 6/6 2024-02-09T21:47:50.515 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-09T21:47:50.515 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:50.515 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-02-09T21:47:50.515 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-09T21:47:50.515 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-02-09T21:47:50.515 INFO:teuthology.orchestra.run.smithi006.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-09T21:47:50.515 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-09T21:47:50.515 INFO:teuthology.orchestra.run.smithi006.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-09T21:47:50.515 INFO:teuthology.orchestra.run.smithi006.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-09T21:47:50.516 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:50.516 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-02-09T21:47:50.766 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-fuse 2024-02-09T21:47:51.363 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:29 ago on Fri 09 Feb 2024 09:46:22 PM UTC. 2024-02-09T21:47:51.530 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-02-09T21:47:51.530 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:47:51.530 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-02-09T21:47:51.531 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:47:51.531 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-02-09T21:47:51.531 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 812 k 2024-02-09T21:47:51.531 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-02-09T21:47:51.531 INFO:teuthology.orchestra.run.smithi006.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-09T21:47:51.531 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:51.531 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-02-09T21:47:51.531 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:47:51.531 INFO:teuthology.orchestra.run.smithi006.stdout:Install 2 Packages 2024-02-09T21:47:51.531 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:51.531 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 891 k 2024-02-09T21:47:51.531 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 2.6 M 2024-02-09T21:47:51.531 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-02-09T21:47:51.644 INFO:teuthology.orchestra.run.smithi006.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 696 kB/s | 79 kB 00:00 2024-02-09T21:47:51.678 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 6/6 2024-02-09T21:47:51.678 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 1/6 2024-02-09T21:47:51.678 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-09T21:47:51.678 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-02-09T21:47:51.678 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-02-09T21:47:51.678 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-09T21:47:51.878 INFO:teuthology.orchestra.run.smithi006.stdout:(2/2): ceph-fuse-17.2.7-505.g72ce7c1d.el9.x86_6 2.3 MB/s | 812 kB 00:00 2024-02-09T21:47:51.879 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:47:51.879 INFO:teuthology.orchestra.run.smithi006.stdout:Total 2.5 MB/s | 891 kB 00:00 2024-02-09T21:47:51.880 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-02-09T21:47:51.902 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-02-09T21:47:51.902 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-02-09T21:47:51.980 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-02-09T21:47:51.981 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-02-09T21:47:51.987 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-09T21:47:51.987 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:51.987 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-09T21:47:51.987 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm-2:17.2.7-505.g72ce7c1d.el9.noarch 2024-02-09T21:47:51.987 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-02-09T21:47:51.988 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-09T21:47:51.988 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-09T21:47:51.988 INFO:teuthology.orchestra.run.smithi090.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-09T21:47:51.988 INFO:teuthology.orchestra.run.smithi090.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-09T21:47:51.988 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:51.988 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-09T21:47:52.167 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-02-09T21:47:52.270 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-fuse 2024-02-09T21:47:52.294 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-09T21:47:52.364 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/2 2024-02-09T21:47:52.850 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:34 ago on Fri 09 Feb 2024 09:46:18 PM UTC. 2024-02-09T21:47:53.008 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-09T21:47:53.008 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:47:53.009 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-02-09T21:47:53.009 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:47:53.009 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-09T21:47:53.009 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 812 k 2024-02-09T21:47:53.009 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-02-09T21:47:53.009 INFO:teuthology.orchestra.run.smithi090.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-09T21:47:53.009 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:53.009 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-09T21:47:53.009 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:47:53.009 INFO:teuthology.orchestra.run.smithi090.stdout:Install 2 Packages 2024-02-09T21:47:53.009 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:53.010 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 891 k 2024-02-09T21:47:53.010 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 2.6 M 2024-02-09T21:47:53.010 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-09T21:47:53.035 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/2 2024-02-09T21:47:53.035 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/2 2024-02-09T21:47:53.148 INFO:teuthology.orchestra.run.smithi090.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 569 kB/s | 79 kB 00:00 2024-02-09T21:47:53.390 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-09T21:47:53.390 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:53.390 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-02-09T21:47:53.391 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-09T21:47:53.391 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:53.391 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-02-09T21:47:53.440 INFO:teuthology.orchestra.run.smithi090.stdout:(2/2): ceph-fuse-17.2.7-505.g72ce7c1d.el9.x86_6 1.8 MB/s | 812 kB 00:00 2024-02-09T21:47:53.441 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:47:53.441 INFO:teuthology.orchestra.run.smithi090.stdout:Total 2.0 MB/s | 891 kB 00:00 2024-02-09T21:47:53.441 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-09T21:47:53.462 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-09T21:47:53.462 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-09T21:47:53.536 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-09T21:47:53.536 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-09T21:47:53.659 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install librados-devel 2024-02-09T21:47:53.710 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-09T21:47:53.845 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-09T21:47:53.907 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/2 2024-02-09T21:47:54.262 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:32 ago on Fri 09 Feb 2024 09:46:22 PM UTC. 2024-02-09T21:47:54.428 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-02-09T21:47:54.428 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:47:54.428 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-02-09T21:47:54.428 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:47:54.429 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-02-09T21:47:54.429 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 126 k 2024-02-09T21:47:54.429 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:54.429 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-02-09T21:47:54.429 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:47:54.429 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-02-09T21:47:54.429 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:54.429 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 126 k 2024-02-09T21:47:54.429 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 448 k 2024-02-09T21:47:54.429 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-02-09T21:47:54.696 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 2/2 2024-02-09T21:47:54.696 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/2 2024-02-09T21:47:54.790 INFO:teuthology.orchestra.run.smithi006.stdout:librados-devel-17.2.7-505.g72ce7c1d.el9.x86_64. 350 kB/s | 126 kB 00:00 2024-02-09T21:47:54.791 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:47:54.791 INFO:teuthology.orchestra.run.smithi006.stdout:Total 349 kB/s | 126 kB 00:00 2024-02-09T21:47:54.791 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-02-09T21:47:54.797 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-02-09T21:47:54.797 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-02-09T21:47:54.854 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-02-09T21:47:54.855 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-02-09T21:47:54.997 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-02-09T21:47:55.024 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-09T21:47:55.024 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:55.024 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-09T21:47:55.024 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-09T21:47:55.024 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:55.024 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-09T21:47:55.100 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librados-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-09T21:47:55.363 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librados-devel 2024-02-09T21:47:55.548 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-09T21:47:55.850 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-09T21:47:55.851 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:55.851 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-02-09T21:47:55.851 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:47:55.851 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:55.851 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-02-09T21:47:55.955 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:37 ago on Fri 09 Feb 2024 09:46:18 PM UTC. 2024-02-09T21:47:56.085 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install libcephfs2 2024-02-09T21:47:56.118 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-09T21:47:56.119 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:47:56.119 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-02-09T21:47:56.119 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:47:56.119 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-09T21:47:56.120 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 126 k 2024-02-09T21:47:56.120 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:56.120 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-09T21:47:56.120 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:47:56.120 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-02-09T21:47:56.120 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:56.120 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 126 k 2024-02-09T21:47:56.120 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 448 k 2024-02-09T21:47:56.120 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-09T21:47:56.375 INFO:teuthology.orchestra.run.smithi090.stdout:librados-devel-17.2.7-505.g72ce7c1d.el9.x86_64. 496 kB/s | 126 kB 00:00 2024-02-09T21:47:56.376 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:47:56.376 INFO:teuthology.orchestra.run.smithi090.stdout:Total 493 kB/s | 126 kB 00:00 2024-02-09T21:47:56.376 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-09T21:47:56.381 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-09T21:47:56.381 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-09T21:47:56.435 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-09T21:47:56.436 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-09T21:47:56.574 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-09T21:47:56.672 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librados-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-09T21:47:56.680 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:34 ago on Fri 09 Feb 2024 09:46:22 PM UTC. 2024-02-09T21:47:56.791 INFO:teuthology.orchestra.run.smithi006.stdout:Package libcephfs2-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-09T21:47:56.848 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-02-09T21:47:56.849 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-02-09T21:47:56.849 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-02-09T21:47:56.914 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install libcephfs-devel 2024-02-09T21:47:57.207 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-09T21:47:57.513 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:35 ago on Fri 09 Feb 2024 09:46:22 PM UTC. 2024-02-09T21:47:57.525 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-09T21:47:57.525 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:57.525 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-09T21:47:57.525 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:47:57.525 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:57.525 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-09T21:47:57.678 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-02-09T21:47:57.679 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:47:57.679 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-02-09T21:47:57.679 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:47:57.679 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-02-09T21:47:57.679 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-devel x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 25 k 2024-02-09T21:47:57.679 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:57.679 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-02-09T21:47:57.679 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:47:57.680 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-02-09T21:47:57.680 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:57.680 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 25 k 2024-02-09T21:47:57.680 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 110 k 2024-02-09T21:47:57.680 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-02-09T21:47:57.873 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install libcephfs2 2024-02-09T21:47:57.894 INFO:teuthology.orchestra.run.smithi006.stdout:libcephfs-devel-17.2.7-505.g72ce7c1d.el9.x86_64 116 kB/s | 25 kB 00:00 2024-02-09T21:47:57.894 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:47:57.895 INFO:teuthology.orchestra.run.smithi006.stdout:Total 116 kB/s | 25 kB 00:00 2024-02-09T21:47:57.895 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-02-09T21:47:57.898 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-02-09T21:47:57.898 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-02-09T21:47:57.900 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-02-09T21:47:57.900 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-02-09T21:47:57.990 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-02-09T21:47:58.111 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephfs-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-09T21:47:58.460 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:40 ago on Fri 09 Feb 2024 09:46:18 PM UTC. 2024-02-09T21:47:58.513 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-09T21:47:58.570 INFO:teuthology.orchestra.run.smithi090.stdout:Package libcephfs2-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-09T21:47:58.624 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-09T21:47:58.625 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-02-09T21:47:58.625 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-09T21:47:58.688 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install libcephfs-devel 2024-02-09T21:47:58.791 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephfs-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-09T21:47:58.792 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:58.792 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-02-09T21:47:58.792 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:47:58.792 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:47:58.792 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-02-09T21:47:58.968 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install librados2 2024-02-09T21:47:59.270 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:41 ago on Fri 09 Feb 2024 09:46:18 PM UTC. 2024-02-09T21:47:59.427 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-09T21:47:59.428 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:47:59.428 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-02-09T21:47:59.428 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:47:59.428 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-09T21:47:59.429 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 25 k 2024-02-09T21:47:59.429 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:59.429 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-09T21:47:59.429 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:47:59.429 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-02-09T21:47:59.429 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:47:59.429 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 25 k 2024-02-09T21:47:59.429 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 110 k 2024-02-09T21:47:59.429 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-09T21:47:59.565 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:37 ago on Fri 09 Feb 2024 09:46:22 PM UTC. 2024-02-09T21:47:59.651 INFO:teuthology.orchestra.run.smithi090.stdout:libcephfs-devel-17.2.7-505.g72ce7c1d.el9.x86_64 113 kB/s | 25 kB 00:00 2024-02-09T21:47:59.651 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:47:59.652 INFO:teuthology.orchestra.run.smithi090.stdout:Total 112 kB/s | 25 kB 00:00 2024-02-09T21:47:59.652 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-09T21:47:59.655 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-09T21:47:59.655 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-09T21:47:59.656 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-09T21:47:59.657 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-09T21:47:59.675 INFO:teuthology.orchestra.run.smithi006.stdout:Package librados2-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-09T21:47:59.731 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-02-09T21:47:59.732 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-02-09T21:47:59.732 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-02-09T21:47:59.745 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-09T21:47:59.791 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install librbd1 2024-02-09T21:47:59.866 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-09T21:48:00.298 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-09T21:48:00.388 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:38 ago on Fri 09 Feb 2024 09:46:22 PM UTC. 2024-02-09T21:48:00.497 INFO:teuthology.orchestra.run.smithi006.stdout:Package librbd1-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-09T21:48:00.552 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-02-09T21:48:00.553 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-02-09T21:48:00.553 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-02-09T21:48:00.570 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-09T21:48:00.570 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:48:00.570 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-09T21:48:00.570 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:48:00.570 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:48:00.570 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-09T21:48:00.616 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-rados 2024-02-09T21:48:00.774 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librados2 2024-02-09T21:48:01.205 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:39 ago on Fri 09 Feb 2024 09:46:22 PM UTC. 2024-02-09T21:48:01.317 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-rados-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-09T21:48:01.357 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:43 ago on Fri 09 Feb 2024 09:46:18 PM UTC. 2024-02-09T21:48:01.372 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-02-09T21:48:01.373 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-02-09T21:48:01.374 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-02-09T21:48:01.441 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-rgw 2024-02-09T21:48:01.463 INFO:teuthology.orchestra.run.smithi090.stdout:Package librados2-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-09T21:48:01.514 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-09T21:48:01.515 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-02-09T21:48:01.516 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-09T21:48:01.627 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librbd1 2024-02-09T21:48:02.033 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:40 ago on Fri 09 Feb 2024 09:46:22 PM UTC. 2024-02-09T21:48:02.143 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-rgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-09T21:48:02.199 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-02-09T21:48:02.200 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-02-09T21:48:02.201 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-02-09T21:48:02.210 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:44 ago on Fri 09 Feb 2024 09:46:18 PM UTC. 2024-02-09T21:48:02.270 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-cephfs 2024-02-09T21:48:02.316 INFO:teuthology.orchestra.run.smithi090.stdout:Package librbd1-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-09T21:48:02.367 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-09T21:48:02.368 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-02-09T21:48:02.368 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-09T21:48:02.452 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rados 2024-02-09T21:48:02.870 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:40 ago on Fri 09 Feb 2024 09:46:22 PM UTC. 2024-02-09T21:48:02.979 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-cephfs-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-09T21:48:03.029 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:45 ago on Fri 09 Feb 2024 09:46:18 PM UTC. 2024-02-09T21:48:03.034 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-02-09T21:48:03.035 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-02-09T21:48:03.035 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-02-09T21:48:03.099 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-rbd 2024-02-09T21:48:03.135 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rados-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-09T21:48:03.188 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-09T21:48:03.189 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-02-09T21:48:03.189 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-09T21:48:03.244 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rgw 2024-02-09T21:48:03.693 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:41 ago on Fri 09 Feb 2024 09:46:22 PM UTC. 2024-02-09T21:48:03.804 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-rbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-09T21:48:03.828 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:45 ago on Fri 09 Feb 2024 09:46:18 PM UTC. 2024-02-09T21:48:03.860 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-02-09T21:48:03.861 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-02-09T21:48:03.861 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-02-09T21:48:03.920 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install rbd-fuse 2024-02-09T21:48:03.935 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rgw-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-09T21:48:03.988 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-09T21:48:03.989 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-02-09T21:48:03.989 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-09T21:48:04.055 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-cephfs 2024-02-09T21:48:04.520 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:42 ago on Fri 09 Feb 2024 09:46:22 PM UTC. 2024-02-09T21:48:04.639 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:46 ago on Fri 09 Feb 2024 09:46:18 PM UTC. 2024-02-09T21:48:04.685 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-02-09T21:48:04.689 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:48:04.689 INFO:teuthology.orchestra.run.smithi006.stdout: Package Architecture Version Repository Size 2024-02-09T21:48:04.689 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:48:04.689 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-02-09T21:48:04.689 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 81 k 2024-02-09T21:48:04.689 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:48:04.690 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-02-09T21:48:04.690 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:48:04.690 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-02-09T21:48:04.690 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:48:04.690 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 81 k 2024-02-09T21:48:04.690 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 222 k 2024-02-09T21:48:04.690 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-02-09T21:48:04.744 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-cephfs-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-09T21:48:04.800 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-09T21:48:04.801 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-02-09T21:48:04.802 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-09T21:48:04.873 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rbd 2024-02-09T21:48:04.981 INFO:teuthology.orchestra.run.smithi006.stdout:rbd-fuse-17.2.7-505.g72ce7c1d.el9.x86_64.rpm 277 kB/s | 81 kB 00:00 2024-02-09T21:48:04.982 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:48:04.982 INFO:teuthology.orchestra.run.smithi006.stdout:Total 276 kB/s | 81 kB 00:00 2024-02-09T21:48:04.982 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-02-09T21:48:04.989 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-02-09T21:48:04.989 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-02-09T21:48:05.046 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-02-09T21:48:05.047 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-02-09T21:48:05.181 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-02-09T21:48:05.272 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-09T21:48:05.457 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:47 ago on Fri 09 Feb 2024 09:46:18 PM UTC. 2024-02-09T21:48:05.564 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 is already installed. 2024-02-09T21:48:05.617 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-09T21:48:05.618 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-02-09T21:48:05.618 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-09T21:48:05.671 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-09T21:48:05.679 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-fuse 2024-02-09T21:48:05.963 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-09T21:48:05.963 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:48:05.963 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-02-09T21:48:05.963 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:48:05.963 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:48:05.963 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-02-09T21:48:06.175 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install rbd-mirror 2024-02-09T21:48:06.267 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:48 ago on Fri 09 Feb 2024 09:46:18 PM UTC. 2024-02-09T21:48:06.426 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-09T21:48:06.426 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:48:06.426 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-02-09T21:48:06.427 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:48:06.427 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-09T21:48:06.427 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 81 k 2024-02-09T21:48:06.427 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:48:06.427 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-09T21:48:06.427 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:48:06.427 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-02-09T21:48:06.427 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:48:06.427 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 81 k 2024-02-09T21:48:06.427 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 222 k 2024-02-09T21:48:06.427 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-09T21:48:06.657 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-fuse-17.2.7-505.g72ce7c1d.el9.x86_64.rpm 354 kB/s | 81 kB 00:00 2024-02-09T21:48:06.658 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:48:06.658 INFO:teuthology.orchestra.run.smithi090.stdout:Total 352 kB/s | 81 kB 00:00 2024-02-09T21:48:06.658 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-09T21:48:06.665 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-09T21:48:06.665 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-09T21:48:06.720 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-09T21:48:06.720 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-09T21:48:06.771 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:44 ago on Fri 09 Feb 2024 09:46:22 PM UTC. 2024-02-09T21:48:06.853 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-09T21:48:06.940 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-02-09T21:48:06.941 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:48:06.941 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-02-09T21:48:06.941 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:48:06.941 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-02-09T21:48:06.941 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 2.9 M 2024-02-09T21:48:06.941 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:48:06.941 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-02-09T21:48:06.941 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:48:06.941 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-02-09T21:48:06.941 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:48:06.941 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 2.9 M 2024-02-09T21:48:06.941 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 12 M 2024-02-09T21:48:06.942 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-02-09T21:48:06.955 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-09T21:48:07.415 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-09T21:48:07.712 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-09T21:48:07.712 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:48:07.712 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-09T21:48:07.712 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:48:07.713 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:48:07.713 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-09T21:48:07.892 INFO:teuthology.orchestra.run.smithi006.stdout:rbd-mirror-17.2.7-505.g72ce7c1d.el9.x86_64.rpm 3.1 MB/s | 2.9 MB 00:00 2024-02-09T21:48:07.892 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:48:07.893 INFO:teuthology.orchestra.run.smithi006.stdout:Total 3.1 MB/s | 2.9 MB 00:00 2024-02-09T21:48:07.893 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-02-09T21:48:07.903 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-02-09T21:48:07.903 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-02-09T21:48:07.983 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-02-09T21:48:07.983 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-02-09T21:48:08.034 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-mirror 2024-02-09T21:48:08.410 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-02-09T21:48:08.476 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-mirror-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-09T21:48:08.499 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-mirror-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-09T21:48:08.499 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T21:48:08.499 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-09T21:48:08.500 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-09T21:48:08.500 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-09T21:48:08.500 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:48:08.638 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:50 ago on Fri 09 Feb 2024 09:46:18 PM UTC. 2024-02-09T21:48:08.812 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-09T21:48:08.812 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:48:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-02-09T21:48:08.813 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:48:08.813 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-09T21:48:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 2.9 M 2024-02-09T21:48:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:48:08.813 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-09T21:48:08.813 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:48:08.813 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-02-09T21:48:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:48:08.813 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 2.9 M 2024-02-09T21:48:08.813 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 12 M 2024-02-09T21:48:08.813 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-09T21:48:09.628 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-mirror-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-09T21:48:09.628 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:48:09.628 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-02-09T21:48:09.629 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:48:09.629 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:48:09.629 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-02-09T21:48:09.857 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install rbd-nbd 2024-02-09T21:48:09.966 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-mirror-17.2.7-505.g72ce7c1d.el9.x86_64.rpm 2.5 MB/s | 2.9 MB 00:01 2024-02-09T21:48:09.967 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:48:09.967 INFO:teuthology.orchestra.run.smithi090.stdout:Total 2.5 MB/s | 2.9 MB 00:01 2024-02-09T21:48:09.967 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-09T21:48:09.976 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-09T21:48:09.976 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-09T21:48:10.053 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-09T21:48:10.053 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-09T21:48:10.455 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-09T21:48:10.457 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:48 ago on Fri 09 Feb 2024 09:46:22 PM UTC. 2024-02-09T21:48:10.526 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-mirror-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-09T21:48:10.550 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-mirror-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-09T21:48:10.550 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T21:48:10.551 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-09T21:48:10.551 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-09T21:48:10.551 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-09T21:48:10.551 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:48:10.626 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-02-09T21:48:10.626 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:48:10.626 INFO:teuthology.orchestra.run.smithi006.stdout: Package Architecture Version Repository Size 2024-02-09T21:48:10.627 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:48:10.627 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-02-09T21:48:10.627 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 164 k 2024-02-09T21:48:10.627 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:48:10.627 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-02-09T21:48:10.627 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-02-09T21:48:10.627 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-02-09T21:48:10.627 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:48:10.627 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 164 k 2024-02-09T21:48:10.627 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 477 k 2024-02-09T21:48:10.627 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-02-09T21:48:11.187 INFO:teuthology.orchestra.run.smithi006.stdout:rbd-nbd-17.2.7-505.g72ce7c1d.el9.x86_64.rpm 294 kB/s | 164 kB 00:00 2024-02-09T21:48:11.188 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:48:11.188 INFO:teuthology.orchestra.run.smithi006.stdout:Total 293 kB/s | 164 kB 00:00 2024-02-09T21:48:11.189 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-02-09T21:48:11.203 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-02-09T21:48:11.203 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-02-09T21:48:11.270 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-02-09T21:48:11.271 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-02-09T21:48:11.424 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-02-09T21:48:11.537 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-nbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-09T21:48:11.874 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-mirror-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-09T21:48:11.874 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:48:11.874 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-09T21:48:11.874 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:48:11.874 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:48:11.874 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-09T21:48:11.974 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-nbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-09T21:48:12.150 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-nbd 2024-02-09T21:48:12.261 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-nbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-09T21:48:12.261 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:48:12.261 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-02-09T21:48:12.261 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:48:12.261 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:48:12.261 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-02-09T21:48:12.497 DEBUG:teuthology.parallel:result is None 2024-02-09T21:48:12.753 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:54 ago on Fri 09 Feb 2024 09:46:18 PM UTC. 2024-02-09T21:48:12.910 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-09T21:48:12.911 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:48:12.911 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-02-09T21:48:12.911 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:48:12.911 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-09T21:48:12.911 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd x86_64 2:17.2.7-505.g72ce7c1d.el9 ceph 164 k 2024-02-09T21:48:12.911 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:48:12.911 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-09T21:48:12.911 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-09T21:48:12.911 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-02-09T21:48:12.911 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:48:12.912 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 164 k 2024-02-09T21:48:12.912 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 477 k 2024-02-09T21:48:12.912 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-09T21:48:13.232 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-nbd-17.2.7-505.g72ce7c1d.el9.x86_64.rpm 515 kB/s | 164 kB 00:00 2024-02-09T21:48:13.233 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:48:13.233 INFO:teuthology.orchestra.run.smithi090.stdout:Total 512 kB/s | 164 kB 00:00 2024-02-09T21:48:13.233 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-09T21:48:13.241 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-09T21:48:13.241 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-09T21:48:13.296 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-09T21:48:13.296 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-09T21:48:13.436 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-09T21:48:13.540 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-nbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-09T21:48:14.039 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-nbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-09T21:48:14.334 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-nbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 1/1 2024-02-09T21:48:14.334 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:48:14.334 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-09T21:48:14.334 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd-2:17.2.7-505.g72ce7c1d.el9.x86_64 2024-02-09T21:48:14.335 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:48:14.335 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-09T21:48:14.624 DEBUG:teuthology.parallel:result is None 2024-02-09T21:48:14.625 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-09T21:48:14.625 INFO:teuthology.packaging:ref: None 2024-02-09T21:48:14.625 INFO:teuthology.packaging:tag: None 2024-02-09T21:48:14.625 INFO:teuthology.packaging:branch: quincy 2024-02-09T21:48:14.625 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T21:48:14.626 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-09T21:48:14.858 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-09T21:48:14.883 INFO:teuthology.orchestra.run.smithi006.stdout:17.2.7-505.g72ce7c1d.el9 2024-02-09T21:48:14.884 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-505.g72ce7c1d.el9 2024-02-09T21:48:14.884 INFO:teuthology.task.install:The correct ceph version 17.2.7-505.g72ce7c1d is installed. 2024-02-09T21:48:14.885 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-09T21:48:14.886 INFO:teuthology.packaging:ref: None 2024-02-09T21:48:14.886 INFO:teuthology.packaging:tag: None 2024-02-09T21:48:14.886 INFO:teuthology.packaging:branch: quincy 2024-02-09T21:48:14.886 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T21:48:14.886 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-09T21:48:15.135 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-09T21:48:15.156 INFO:teuthology.orchestra.run.smithi090.stdout:17.2.7-505.g72ce7c1d.el9 2024-02-09T21:48:15.156 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-505.g72ce7c1d.el9 2024-02-09T21:48:15.156 INFO:teuthology.task.install:The correct ceph version 17.2.7-505.g72ce7c1d is installed. 2024-02-09T21:48:15.158 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-09T21:48:15.158 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-02-09T21:48:15.158 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-09T21:48:15.192 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-09T21:48:15.192 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-09T21:48:15.224 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-09T21:48:15.224 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-02-09T21:48:15.224 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/daemon-helper 2024-02-09T21:48:15.263 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-09T21:48:15.334 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-09T21:48:15.334 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/daemon-helper 2024-02-09T21:48:15.360 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-09T21:48:15.427 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-09T21:48:15.427 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-02-09T21:48:15.427 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-09T21:48:15.459 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-09T21:48:15.530 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-09T21:48:15.530 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-09T21:48:15.557 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-09T21:48:15.623 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-09T21:48:15.624 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-02-09T21:48:15.624 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/stdin-killer 2024-02-09T21:48:15.654 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-09T21:48:15.726 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-09T21:48:15.726 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/stdin-killer 2024-02-09T21:48:15.752 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-09T21:48:15.820 INFO:teuthology.run_tasks:Running task print... 2024-02-09T21:48:15.832 INFO:teuthology.task.print:**** done install task... 2024-02-09T21:48:15.832 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-09T21:48:15.966 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'} 2024-02-09T21:48:15.966 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-02-09T21:48:15.967 INFO:tasks.cephadm:Cluster fsid is ee26d820-c794-11ee-95b8-87774f69a715 2024-02-09T21:48:15.967 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-09T21:48:15.967 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-02-09T21:48:15.967 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi006': '172.21.15.6', 'mon.smithi090': '172.21.15.90'} 2024-02-09T21:48:15.968 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-09T21:48:15.968 DEBUG:teuthology.orchestra.run.smithi006:> sudo hostname $(hostname -s) 2024-02-09T21:48:15.999 DEBUG:teuthology.orchestra.run.smithi090:> sudo hostname $(hostname -s) 2024-02-09T21:48:16.026 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-02-09T21:48:16.027 DEBUG:teuthology.orchestra.run.smithi006:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-09T21:48:16.276 INFO:teuthology.orchestra.run.smithi006.stdout:-rw-r--r--. 1 ubuntu ubuntu 367151 Feb 9 21:48 /home/ubuntu/cephtest/cephadm 2024-02-09T21:48:16.276 DEBUG:teuthology.orchestra.run.smithi090:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-09T21:48:16.455 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r--. 1 ubuntu ubuntu 367151 Feb 9 21:48 /home/ubuntu/cephtest/cephadm 2024-02-09T21:48:16.455 DEBUG:teuthology.orchestra.run.smithi006:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-09T21:48:16.474 DEBUG:teuthology.orchestra.run.smithi090:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-09T21:48:16.499 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-02-09T21:48:16.499 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-02-09T21:48:16.517 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-02-09T21:48:17.356 INFO:teuthology.orchestra.run.smithi006.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-09T21:48:17.541 INFO:teuthology.orchestra.run.smithi090.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-09T21:49:16.328 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-02-09T21:49:16.328 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_version": "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)", 2024-02-09T21:49:16.328 INFO:teuthology.orchestra.run.smithi006.stdout: "image_id": "a2a0bd126c96301261ac6c5708253ef5b41bd73b23624c6dbd2eed86fc57c9b5", 2024-02-09T21:49:16.328 INFO:teuthology.orchestra.run.smithi006.stdout: "repo_digests": [ 2024-02-09T21:49:16.329 INFO:teuthology.orchestra.run.smithi006.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc" 2024-02-09T21:49:16.329 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-02-09T21:49:16.329 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-02-09T21:49:22.433 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-02-09T21:49:22.433 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)", 2024-02-09T21:49:22.433 INFO:teuthology.orchestra.run.smithi090.stdout: "image_id": "a2a0bd126c96301261ac6c5708253ef5b41bd73b23624c6dbd2eed86fc57c9b5", 2024-02-09T21:49:22.433 INFO:teuthology.orchestra.run.smithi090.stdout: "repo_digests": [ 2024-02-09T21:49:22.433 INFO:teuthology.orchestra.run.smithi090.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc" 2024-02-09T21:49:22.433 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-02-09T21:49:22.433 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-02-09T21:49:22.453 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph 2024-02-09T21:49:22.489 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /etc/ceph 2024-02-09T21:49:22.519 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 777 /etc/ceph 2024-02-09T21:49:22.559 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 777 /etc/ceph 2024-02-09T21:49:22.586 INFO:tasks.cephadm:Writing seed config... 2024-02-09T21:49:22.587 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-02-09T21:49:22.587 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-02-09T21:49:22.587 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-02-09T21:49:22.587 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-02-09T21:49:22.588 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-02-09T21:49:22.588 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-02-09T21:49:22.588 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-02-09T21:49:22.588 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-02-09T21:49:22.588 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-02-09T21:49:22.588 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-09T21:49:22.588 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-09T21:49:22.588 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-02-09T21:49:22.588 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-02-09T21:49:22.588 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-02-09T21:49:22.588 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-02-09T21:49:22.588 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-02-09T21:49:22.588 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-02-09T21:49:22.588 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-02-09T21:49:22.589 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-02-09T21:49:22.589 INFO:tasks.cephadm: override: [client] debug client = 20 2024-02-09T21:49:22.589 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-09T21:49:22.589 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-02-09T21:49:22.589 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-02-09T21:49:22.589 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-02-09T21:49:22.589 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-02-09T21:49:22.589 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-02-09T21:49:22.589 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-02-09T21:49:22.589 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-02-09T21:49:22.589 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-02-09T21:49:22.589 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-02-09T21:49:22.589 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-02-09T21:49:22.589 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-02-09T21:49:22.590 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-02-09T21:49:22.590 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-02-09T21:49:22.590 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-09T21:49:22.590 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-09T21:49:22.590 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-09T21:49:22.590 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-09T21:49:22.590 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-09T21:49:22.590 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-02-09T21:49:22.591 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-02-09T21:49:22.591 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-09T21:49:22.617 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = ee26d820-c794-11ee-95b8-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-02-09T21:49:22.617 DEBUG:teuthology.orchestra.run.smithi006:mon.smithi006> sudo journalctl -f -n 0 -u ceph-ee26d820-c794-11ee-95b8-87774f69a715@mon.smithi006.service 2024-02-09T21:49:22.660 INFO:tasks.cephadm:Bootstrapping... 2024-02-09T21:49:22.660 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid ee26d820-c794-11ee-95b8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.6 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-09T21:49:22.878 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-02-09T21:49:22.878 INFO:teuthology.orchestra.run.smithi006.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', 'ee26d820-c794-11ee-95b8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.6', '--skip-admin-label'] 2024-02-09T21:49:22.903 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-09T21:49:22.904 INFO:teuthology.orchestra.run.smithi006.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-02-09T21:49:22.904 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying podman|docker is present... 2024-02-09T21:49:22.928 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-09T21:49:22.928 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying lvm2 is present... 2024-02-09T21:49:22.928 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying time synchronization is in place... 2024-02-09T21:49:22.936 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-09T21:49:22.936 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-09T21:49:22.942 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-09T21:49:22.942 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout inactive 2024-02-09T21:49:22.948 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout enabled 2024-02-09T21:49:22.955 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout active 2024-02-09T21:49:22.955 INFO:teuthology.orchestra.run.smithi006.stdout:Unit chronyd.service is enabled and running 2024-02-09T21:49:22.955 INFO:teuthology.orchestra.run.smithi006.stdout:Repeating the final host check... 2024-02-09T21:49:22.979 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-09T21:49:22.979 INFO:teuthology.orchestra.run.smithi006.stdout:podman (/bin/podman) version 4.9.3 is present 2024-02-09T21:49:22.979 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl is present 2024-02-09T21:49:22.979 INFO:teuthology.orchestra.run.smithi006.stdout:lvcreate is present 2024-02-09T21:49:22.985 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-09T21:49:22.985 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-09T21:49:22.991 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-09T21:49:22.991 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout inactive 2024-02-09T21:49:22.998 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout enabled 2024-02-09T21:49:23.004 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout active 2024-02-09T21:49:23.004 INFO:teuthology.orchestra.run.smithi006.stdout:Unit chronyd.service is enabled and running 2024-02-09T21:49:23.004 INFO:teuthology.orchestra.run.smithi006.stdout:Host looks OK 2024-02-09T21:49:23.004 INFO:teuthology.orchestra.run.smithi006.stdout:Cluster fsid: ee26d820-c794-11ee-95b8-87774f69a715 2024-02-09T21:49:23.005 INFO:teuthology.orchestra.run.smithi006.stdout:Acquiring lock 140045533248000 on /run/cephadm/ee26d820-c794-11ee-95b8-87774f69a715.lock 2024-02-09T21:49:23.005 INFO:teuthology.orchestra.run.smithi006.stdout:Lock 140045533248000 acquired on /run/cephadm/ee26d820-c794-11ee-95b8-87774f69a715.lock 2024-02-09T21:49:23.005 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying IP 172.21.15.6 port 3300 ... 2024-02-09T21:49:23.005 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying IP 172.21.15.6 port 6789 ... 2024-02-09T21:49:23.005 INFO:teuthology.orchestra.run.smithi006.stdout:Base mon IP(s) is [172.21.15.6:3300, 172.21.15.6:6789], mon addrv is [v2:172.21.15.6:3300,v1:172.21.15.6:6789] 2024-02-09T21:49:23.008 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.6 metric 100 2024-02-09T21:49:23.008 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.6 metric 100 2024-02-09T21:49:23.011 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-02-09T21:49:23.011 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-02-09T21:49:23.011 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-02-09T21:49:23.014 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-09T21:49:23.014 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-02-09T21:49:23.014 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-09T21:49:23.014 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-02-09T21:49:23.014 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-02-09T21:49:23.014 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-09T21:49:23.015 INFO:teuthology.orchestra.run.smithi006.stdout:Mon IP `172.21.15.6` is in CIDR network `172.21.0.0/20` 2024-02-09T21:49:23.015 INFO:teuthology.orchestra.run.smithi006.stdout:Mon IP `172.21.15.6` is in CIDR network `172.21.0.0/20` 2024-02-09T21:49:23.015 INFO:teuthology.orchestra.run.smithi006.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-02-09T21:49:23.015 INFO:teuthology.orchestra.run.smithi006.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-09T21:49:23.016 INFO:teuthology.orchestra.run.smithi006.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-09T21:49:23.518 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stdout a2a0bd126c96301261ac6c5708253ef5b41bd73b23624c6dbd2eed86fc57c9b5 2024-02-09T21:49:23.519 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-09T21:49:23.519 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stderr Getting image source signatures 2024-02-09T21:49:23.519 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stderr Copying blob sha256:58df4e03ba066accf3a7e07426bab5f1c867e27a139801c642bdafb42b9a2d9a 2024-02-09T21:49:23.519 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stderr Copying blob sha256:a61c475ff6e77d421bceb2aff73188667ff70f4da71ee8d2a5b09b5c5d4ca045 2024-02-09T21:49:23.519 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stderr Copying config sha256:a2a0bd126c96301261ac6c5708253ef5b41bd73b23624c6dbd2eed86fc57c9b5 2024-02-09T21:49:23.519 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stderr Writing manifest to image destination 2024-02-09T21:49:24.459 INFO:teuthology.orchestra.run.smithi006.stdout:ceph: stdout ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable) 2024-02-09T21:49:24.459 INFO:teuthology.orchestra.run.smithi006.stdout:Ceph version: ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable) 2024-02-09T21:49:24.459 INFO:teuthology.orchestra.run.smithi006.stdout:Extracting ceph user uid/gid from container image... 2024-02-09T21:49:25.296 INFO:teuthology.orchestra.run.smithi006.stdout:stat: stdout 167 167 2024-02-09T21:49:25.297 INFO:teuthology.orchestra.run.smithi006.stdout:Creating initial keys... 2024-02-09T21:49:26.336 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph-authtool: stdout AQDlncZl4ww8JhAAviCVsOQ+3SySSF+XXzjzvQ== 2024-02-09T21:49:27.324 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph-authtool: stdout AQDmncZl/uMaKRAAwMud+blGsuMGUyRMZ/Kw6A== 2024-02-09T21:49:34.717 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph-authtool: stdout AQDoncZlPgcFJBAAWrhCpaFLDvYac/ZXZmLV5A== 2024-02-09T21:49:34.718 INFO:teuthology.orchestra.run.smithi006.stdout:Creating initial monmap... 2024-02-09T21:49:36.880 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-09T21:49:36.880 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-02-09T21:49:36.881 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to ee26d820-c794-11ee-95b8-87774f69a715 2024-02-09T21:49:36.881 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-09T21:49:36.881 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool for smithi006 [v2:172.21.15.6:3300,v1:172.21.15.6:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-09T21:49:36.881 INFO:teuthology.orchestra.run.smithi006.stdout:setting min_mon_release = octopus 2024-02-09T21:49:36.881 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/monmaptool: set fsid to ee26d820-c794-11ee-95b8-87774f69a715 2024-02-09T21:49:36.881 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-09T21:49:36.881 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:49:36.881 INFO:teuthology.orchestra.run.smithi006.stdout:Creating mon... 2024-02-09T21:49:37.771 INFO:teuthology.orchestra.run.smithi006.stdout:create mon.smithi006 on 2024-02-09T21:49:37.978 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-02-09T21:49:38.120 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-09T21:49:38.307 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-ee26d820-c794-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-ee26d820-c794-11ee-95b8-87774f69a715.target. 2024-02-09T21:49:38.307 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-ee26d820-c794-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-ee26d820-c794-11ee-95b8-87774f69a715.target. 2024-02-09T21:49:38.480 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-ee26d820-c794-11ee-95b8-87774f69a715@mon.smithi006 2024-02-09T21:49:38.481 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Failed to reset failed state of unit ceph-ee26d820-c794-11ee-95b8-87774f69a715@mon.smithi006.service: Unit ceph-ee26d820-c794-11ee-95b8-87774f69a715@mon.smithi006.service not loaded. 2024-02-09T21:49:38.658 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-ee26d820-c794-11ee-95b8-87774f69a715.target.wants/ceph-ee26d820-c794-11ee-95b8-87774f69a715@mon.smithi006.service → /etc/systemd/system/ceph-ee26d820-c794-11ee-95b8-87774f69a715@.service. 2024-02-09T21:49:39.196 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-09T21:49:39.196 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout disabled 2024-02-09T21:49:39.204 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-09T21:49:39.204 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout inactive 2024-02-09T21:49:39.204 INFO:teuthology.orchestra.run.smithi006.stdout:firewalld.service is not enabled 2024-02-09T21:49:39.204 INFO:teuthology.orchestra.run.smithi006.stdout:Not possible to enable service . firewalld.service is not available 2024-02-09T21:49:39.205 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for mon to start... 2024-02-09T21:49:39.205 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for mon... 2024-02-09T21:49:40.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:40 smithi006 ceph-mon[28884]: from='client.? 172.21.15.6:0/3427991604' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-09T21:49:40.773 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout cluster: 2024-02-09T21:49:40.779 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout id: ee26d820-c794-11ee-95b8-87774f69a715 2024-02-09T21:49:40.779 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-02-09T21:49:40.779 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-02-09T21:49:40.779 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout services: 2024-02-09T21:49:40.779 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi006 (age 0.595278s) 2024-02-09T21:49:40.779 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-02-09T21:49:40.779 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-02-09T21:49:40.780 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-02-09T21:49:40.780 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout data: 2024-02-09T21:49:40.780 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-02-09T21:49:40.780 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-02-09T21:49:40.780 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-02-09T21:49:40.780 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout pgs: 2024-02-09T21:49:40.780 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-02-09T21:49:40.780 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.651+0000 7f80a57c0700 1 Processor -- start 2024-02-09T21:49:40.780 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.651+0000 7f80a57c0700 1 -- start start 2024-02-09T21:49:40.780 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.652+0000 7f80a57c0700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80a0103d80 0x7f80a0104190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:49:40.780 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.652+0000 7f80a57c0700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a01046d0 con 0x7f80a0103d80 2024-02-09T21:49:40.780 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.652+0000 7f809effd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80a0103d80 0x7f80a0104190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:49:40.780 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.652+0000 7f809effd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80a0103d80 0x7f80a0104190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42324/0 (socket says 172.21.15.6:42324) 2024-02-09T21:49:40.780 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.652+0000 7f809effd700 1 -- 172.21.15.6:0/627202336 learned_addr learned my addr 172.21.15.6:0/627202336 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:49:40.780 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.652+0000 7f809effd700 1 -- 172.21.15.6:0/627202336 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a0104810 con 0x7f80a0103d80 2024-02-09T21:49:40.780 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.652+0000 7f809effd700 1 --2- 172.21.15.6:0/627202336 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80a0103d80 0x7f80a0104190 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f80900098e0 tx=0x7f8090009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e59ae205a071e3be server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:49:40.781 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.653+0000 7f809dffb700 1 -- 172.21.15.6:0/627202336 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f809000b860 con 0x7f80a0103d80 2024-02-09T21:49:40.781 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.653+0000 7f809dffb700 1 -- 172.21.15.6:0/627202336 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809000b9c0 con 0x7f80a0103d80 2024-02-09T21:49:40.781 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.653+0000 7f809dffb700 1 -- 172.21.15.6:0/627202336 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f809000bc90 con 0x7f80a0103d80 2024-02-09T21:49:40.781 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.653+0000 7f80a57c0700 1 -- 172.21.15.6:0/627202336 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80a0103d80 msgr2=0x7f80a0104190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:49:40.781 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.653+0000 7f80a57c0700 1 --2- 172.21.15.6:0/627202336 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80a0103d80 0x7f80a0104190 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f80900098e0 tx=0x7f8090009250 comp rx=0 tx=0).stop 2024-02-09T21:49:40.781 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.653+0000 7f80a57c0700 1 -- 172.21.15.6:0/627202336 shutdown_connections 2024-02-09T21:49:40.781 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.653+0000 7f80a57c0700 1 --2- 172.21.15.6:0/627202336 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80a0103d80 0x7f80a0104190 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:49:40.781 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.653+0000 7f80a57c0700 1 -- 172.21.15.6:0/627202336 >> 172.21.15.6:0/627202336 conn(0x7f80a00fc580 msgr2=0x7f80a00fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:49:40.781 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.653+0000 7f80a57c0700 1 -- 172.21.15.6:0/627202336 shutdown_connections 2024-02-09T21:49:40.781 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.653+0000 7f80a57c0700 1 -- 172.21.15.6:0/627202336 wait complete. 2024-02-09T21:49:40.781 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.654+0000 7f80a57c0700 1 Processor -- start 2024-02-09T21:49:40.781 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.654+0000 7f80a57c0700 1 -- start start 2024-02-09T21:49:40.781 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.654+0000 7f80a57c0700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80a0103d80 0x7f80a0111060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:49:40.781 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.654+0000 7f80a57c0700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a01115a0 con 0x7f80a0103d80 2024-02-09T21:49:40.781 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.654+0000 7f809effd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80a0103d80 0x7f80a0111060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:49:40.782 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.654+0000 7f809effd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80a0103d80 0x7f80a0111060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42336/0 (socket says 172.21.15.6:42336) 2024-02-09T21:49:40.782 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.654+0000 7f809effd700 1 -- 172.21.15.6:0/3427991604 learned_addr learned my addr 172.21.15.6:0/3427991604 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:49:40.782 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.654+0000 7f809effd700 1 -- 172.21.15.6:0/3427991604 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80900094f0 con 0x7f80a0103d80 2024-02-09T21:49:40.782 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.655+0000 7f809effd700 1 --2- 172.21.15.6:0/3427991604 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80a0103d80 0x7f80a0111060 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8090003080 tx=0x7f8090016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:49:40.782 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.655+0000 7f80877fe700 1 -- 172.21.15.6:0/3427991604 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f809001c5c0 con 0x7f80a0103d80 2024-02-09T21:49:40.782 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.655+0000 7f80877fe700 1 -- 172.21.15.6:0/3427991604 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809001c720 con 0x7f80a0103d80 2024-02-09T21:49:40.782 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.655+0000 7f80a57c0700 1 -- 172.21.15.6:0/3427991604 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a01117a0 con 0x7f80a0103d80 2024-02-09T21:49:40.782 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.655+0000 7f80877fe700 1 -- 172.21.15.6:0/3427991604 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f809001ca30 con 0x7f80a0103d80 2024-02-09T21:49:40.782 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.655+0000 7f80a57c0700 1 -- 172.21.15.6:0/3427991604 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a0112440 con 0x7f80a0103d80 2024-02-09T21:49:40.782 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.655+0000 7f80877fe700 1 -- 172.21.15.6:0/3427991604 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f809001a450 con 0x7f80a0103d80 2024-02-09T21:49:40.782 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.655+0000 7f80877fe700 1 -- 172.21.15.6:0/3427991604 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f809002e790 con 0x7f80a0103d80 2024-02-09T21:49:40.782 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.656+0000 7f80a57c0700 1 -- 172.21.15.6:0/3427991604 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80a005f5c0 con 0x7f80a0103d80 2024-02-09T21:49:40.782 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.658+0000 7f80877fe700 1 -- 172.21.15.6:0/3427991604 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f809001a890 con 0x7f80a0103d80 2024-02-09T21:49:40.782 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.731+0000 7f80a57c0700 1 -- 172.21.15.6:0/3427991604 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f80a0064ad0 con 0x7f80a0103d80 2024-02-09T21:49:40.782 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.731+0000 7f80877fe700 1 -- 172.21.15.6:0/3427991604 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f809001aa30 con 0x7f80a0103d80 2024-02-09T21:49:40.782 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.732+0000 7f80a57c0700 1 -- 172.21.15.6:0/3427991604 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80a0103d80 msgr2=0x7f80a0111060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:49:40.783 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.732+0000 7f80a57c0700 1 --2- 172.21.15.6:0/3427991604 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80a0103d80 0x7f80a0111060 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8090003080 tx=0x7f8090016000 comp rx=0 tx=0).stop 2024-02-09T21:49:40.783 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.732+0000 7f80a57c0700 1 -- 172.21.15.6:0/3427991604 shutdown_connections 2024-02-09T21:49:40.783 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.732+0000 7f80a57c0700 1 --2- 172.21.15.6:0/3427991604 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80a0103d80 0x7f80a0111060 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:49:40.783 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.732+0000 7f80a57c0700 1 -- 172.21.15.6:0/3427991604 >> 172.21.15.6:0/3427991604 conn(0x7f80a00fc580 msgr2=0x7f80a0107e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:49:40.783 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.732+0000 7f80a57c0700 1 -- 172.21.15.6:0/3427991604 shutdown_connections 2024-02-09T21:49:40.783 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:39.732+0000 7f80a57c0700 1 -- 172.21.15.6:0/3427991604 wait complete. 2024-02-09T21:49:40.783 INFO:teuthology.orchestra.run.smithi006.stdout:mon is available 2024-02-09T21:49:40.783 INFO:teuthology.orchestra.run.smithi006.stdout:Assimilating anything we can from ceph.conf... 2024-02-09T21:49:41.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:41 smithi006 ceph-mon[28884]: from='client.? 172.21.15.6:0/2977667293' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-09T21:49:41.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:41 smithi006 ceph-mon[28884]: from='client.? 172.21.15.6:0/2977667293' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-09T21:49:43.210 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-02-09T21:49:43.210 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout [global] 2024-02-09T21:49:43.210 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout fsid = ee26d820-c794-11ee-95b8-87774f69a715 2024-02-09T21:49:43.210 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.6:3300,v1:172.21.15.6:6789] 2024-02-09T21:49:43.210 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-09T21:49:43.210 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-09T21:49:43.210 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-09T21:49:43.211 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-09T21:49:43.211 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-02-09T21:49:43.211 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-09T21:49:43.211 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-09T21:49:43.211 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-02-09T21:49:43.211 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout [osd] 2024-02-09T21:49:43.211 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-02-09T21:49:43.211 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-02-09T21:49:43.211 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-02-09T21:49:43.211 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-09T21:49:43.211 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-09T21:49:43.211 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.194+0000 7f849f96c700 1 Processor -- start 2024-02-09T21:49:43.211 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.194+0000 7f849f96c700 1 -- start start 2024-02-09T21:49:43.211 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.194+0000 7f849f96c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8498076c20 0x7f8498105380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:49:43.212 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.194+0000 7f849f96c700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84981058c0 con 0x7f8498076c20 2024-02-09T21:49:43.212 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.194+0000 7f849d708700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8498076c20 0x7f8498105380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:49:43.212 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.194+0000 7f849d708700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8498076c20 0x7f8498105380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42350/0 (socket says 172.21.15.6:42350) 2024-02-09T21:49:43.212 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.194+0000 7f849d708700 1 -- 172.21.15.6:0/3418749545 learned_addr learned my addr 172.21.15.6:0/3418749545 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:49:43.212 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.194+0000 7f849d708700 1 -- 172.21.15.6:0/3418749545 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8498105a00 con 0x7f8498076c20 2024-02-09T21:49:43.212 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.195+0000 7f849d708700 1 --2- 172.21.15.6:0/3418749545 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8498076c20 0x7f8498105380 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f84940098e0 tx=0x7f8494009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14c83a6e77c7c669 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:49:43.212 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.195+0000 7f848ffff700 1 -- 172.21.15.6:0/3418749545 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f849400b860 con 0x7f8498076c20 2024-02-09T21:49:43.212 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.195+0000 7f848ffff700 1 -- 172.21.15.6:0/3418749545 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f849400b9c0 con 0x7f8498076c20 2024-02-09T21:49:43.212 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.195+0000 7f848ffff700 1 -- 172.21.15.6:0/3418749545 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f849400bc90 con 0x7f8498076c20 2024-02-09T21:49:43.212 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.195+0000 7f849f96c700 1 -- 172.21.15.6:0/3418749545 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8498076c20 msgr2=0x7f8498105380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:49:43.212 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.195+0000 7f849f96c700 1 --2- 172.21.15.6:0/3418749545 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8498076c20 0x7f8498105380 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f84940098e0 tx=0x7f8494009250 comp rx=0 tx=0).stop 2024-02-09T21:49:43.212 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.195+0000 7f849f96c700 1 -- 172.21.15.6:0/3418749545 shutdown_connections 2024-02-09T21:49:43.212 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.195+0000 7f849f96c700 1 --2- 172.21.15.6:0/3418749545 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8498076c20 0x7f8498105380 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:49:43.213 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.195+0000 7f849f96c700 1 -- 172.21.15.6:0/3418749545 >> 172.21.15.6:0/3418749545 conn(0x7f84980fcdd0 msgr2=0x7f84980ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:49:43.213 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.196+0000 7f849f96c700 1 -- 172.21.15.6:0/3418749545 shutdown_connections 2024-02-09T21:49:43.213 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.196+0000 7f849f96c700 1 -- 172.21.15.6:0/3418749545 wait complete. 2024-02-09T21:49:43.213 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.196+0000 7f849f96c700 1 Processor -- start 2024-02-09T21:49:43.213 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.196+0000 7f849f96c700 1 -- start start 2024-02-09T21:49:43.213 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.196+0000 7f849f96c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8498076c20 0x7f84981110a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:49:43.213 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.196+0000 7f849f96c700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84981115e0 con 0x7f8498076c20 2024-02-09T21:49:43.213 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.196+0000 7f849d708700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8498076c20 0x7f84981110a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:49:43.213 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.196+0000 7f849d708700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8498076c20 0x7f84981110a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42358/0 (socket says 172.21.15.6:42358) 2024-02-09T21:49:43.213 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.196+0000 7f849d708700 1 -- 172.21.15.6:0/2977667293 learned_addr learned my addr 172.21.15.6:0/2977667293 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:49:43.213 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.197+0000 7f849d708700 1 -- 172.21.15.6:0/2977667293 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84940094f0 con 0x7f8498076c20 2024-02-09T21:49:43.213 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.197+0000 7f849d708700 1 --2- 172.21.15.6:0/2977667293 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8498076c20 0x7f84981110a0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f8494009290 tx=0x7f8494016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:49:43.213 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.197+0000 7f848dffb700 1 -- 172.21.15.6:0/2977667293 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f849401c5c0 con 0x7f8498076c20 2024-02-09T21:49:43.213 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.197+0000 7f848dffb700 1 -- 172.21.15.6:0/2977667293 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f849401c740 con 0x7f8498076c20 2024-02-09T21:49:43.213 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.197+0000 7f849f96c700 1 -- 172.21.15.6:0/2977667293 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84981117e0 con 0x7f8498076c20 2024-02-09T21:49:43.214 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.197+0000 7f848dffb700 1 -- 172.21.15.6:0/2977667293 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f849401ca50 con 0x7f8498076c20 2024-02-09T21:49:43.214 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.197+0000 7f849f96c700 1 -- 172.21.15.6:0/2977667293 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8498112480 con 0x7f8498076c20 2024-02-09T21:49:43.214 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.197+0000 7f848dffb700 1 -- 172.21.15.6:0/2977667293 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f849401a450 con 0x7f8498076c20 2024-02-09T21:49:43.214 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.197+0000 7f848dffb700 1 -- 172.21.15.6:0/2977667293 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f849402e790 con 0x7f8498076c20 2024-02-09T21:49:43.214 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.198+0000 7f849f96c700 1 -- 172.21.15.6:0/2977667293 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f849805f5c0 con 0x7f8498076c20 2024-02-09T21:49:43.214 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.200+0000 7f848dffb700 1 -- 172.21.15.6:0/2977667293 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f849401ce30 con 0x7f8498076c20 2024-02-09T21:49:43.214 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.269+0000 7f849f96c700 1 -- 172.21.15.6:0/2977667293 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f849810ab40 con 0x7f8498076c20 2024-02-09T21:49:43.214 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.273+0000 7f848dffb700 1 -- 172.21.15.6:0/2977667293 <== mon.0 v2:172.21.15.6:3300/0 7 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f849402e9f0 con 0x7f8498076c20 2024-02-09T21:49:43.214 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.273+0000 7f848dffb700 1 -- 172.21.15.6:0/2977667293 <== mon.0 v2:172.21.15.6:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+427 (secure 0 0 0) 0x7f849401b740 con 0x7f8498076c20 2024-02-09T21:49:43.214 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.274+0000 7f849f96c700 1 -- 172.21.15.6:0/2977667293 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8498076c20 msgr2=0x7f84981110a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:49:43.214 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.274+0000 7f849f96c700 1 --2- 172.21.15.6:0/2977667293 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8498076c20 0x7f84981110a0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f8494009290 tx=0x7f8494016000 comp rx=0 tx=0).stop 2024-02-09T21:49:43.214 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.274+0000 7f849f96c700 1 -- 172.21.15.6:0/2977667293 shutdown_connections 2024-02-09T21:49:43.214 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.274+0000 7f849f96c700 1 --2- 172.21.15.6:0/2977667293 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8498076c20 0x7f84981110a0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:49:43.214 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.274+0000 7f849f96c700 1 -- 172.21.15.6:0/2977667293 >> 172.21.15.6:0/2977667293 conn(0x7f84980fcdd0 msgr2=0x7f84980fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:49:43.215 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.274+0000 7f849f96c700 1 -- 172.21.15.6:0/2977667293 shutdown_connections 2024-02-09T21:49:43.215 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:41.274+0000 7f849f96c700 1 -- 172.21.15.6:0/2977667293 wait complete. 2024-02-09T21:49:43.215 INFO:teuthology.orchestra.run.smithi006.stdout:Generating new minimal ceph.conf... 2024-02-09T21:49:43.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:43 smithi006 ceph-mon[28884]: from='client.? 172.21.15.6:0/2685719780' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:49:44.263 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.633+0000 7fb5c26eb700 1 Processor -- start 2024-02-09T21:49:44.263 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.633+0000 7fb5c26eb700 1 -- start start 2024-02-09T21:49:44.263 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.633+0000 7fb5c26eb700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb5bc078d20 0x7fb5bc077230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:49:44.263 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.633+0000 7fb5c26eb700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5bc079130 con 0x7fb5bc078d20 2024-02-09T21:49:44.263 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.633+0000 7fb5bbfff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb5bc078d20 0x7fb5bc077230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:49:44.263 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.634+0000 7fb5bbfff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb5bc078d20 0x7fb5bc077230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42366/0 (socket says 172.21.15.6:42366) 2024-02-09T21:49:44.263 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.634+0000 7fb5bbfff700 1 -- 172.21.15.6:0/473069832 learned_addr learned my addr 172.21.15.6:0/473069832 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:49:44.263 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.634+0000 7fb5bbfff700 1 -- 172.21.15.6:0/473069832 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5bc077770 con 0x7fb5bc078d20 2024-02-09T21:49:44.263 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.634+0000 7fb5bbfff700 1 --2- 172.21.15.6:0/473069832 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb5bc078d20 0x7fb5bc077230 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac0098e0 tx=0x7fb5ac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f055e621ed9bb47d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:49:44.263 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.634+0000 7fb5baffd700 1 -- 172.21.15.6:0/473069832 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5ac00b860 con 0x7fb5bc078d20 2024-02-09T21:49:44.263 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.634+0000 7fb5baffd700 1 -- 172.21.15.6:0/473069832 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb5ac00be50 con 0x7fb5bc078d20 2024-02-09T21:49:44.264 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.634+0000 7fb5baffd700 1 -- 172.21.15.6:0/473069832 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5ac010380 con 0x7fb5bc078d20 2024-02-09T21:49:44.264 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.635+0000 7fb5c26eb700 1 -- 172.21.15.6:0/473069832 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb5bc078d20 msgr2=0x7fb5bc077230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:49:44.264 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.635+0000 7fb5c26eb700 1 --2- 172.21.15.6:0/473069832 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb5bc078d20 0x7fb5bc077230 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac0098e0 tx=0x7fb5ac009250 comp rx=0 tx=0).stop 2024-02-09T21:49:44.264 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.635+0000 7fb5c26eb700 1 -- 172.21.15.6:0/473069832 shutdown_connections 2024-02-09T21:49:44.264 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.635+0000 7fb5c26eb700 1 --2- 172.21.15.6:0/473069832 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb5bc078d20 0x7fb5bc077230 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:49:44.264 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.635+0000 7fb5c26eb700 1 -- 172.21.15.6:0/473069832 >> 172.21.15.6:0/473069832 conn(0x7fb5bc0fded0 msgr2=0x7fb5bc100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:49:44.264 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.635+0000 7fb5c26eb700 1 -- 172.21.15.6:0/473069832 shutdown_connections 2024-02-09T21:49:44.264 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.635+0000 7fb5c26eb700 1 -- 172.21.15.6:0/473069832 wait complete. 2024-02-09T21:49:44.264 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.635+0000 7fb5c26eb700 1 Processor -- start 2024-02-09T21:49:44.264 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.636+0000 7fb5c26eb700 1 -- start start 2024-02-09T21:49:44.264 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.636+0000 7fb5c26eb700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb5bc078d20 0x7fb5bc113810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:49:44.264 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.636+0000 7fb5c26eb700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5bc113d50 con 0x7fb5bc078d20 2024-02-09T21:49:44.264 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.636+0000 7fb5bbfff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb5bc078d20 0x7fb5bc113810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:49:44.264 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.636+0000 7fb5bbfff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb5bc078d20 0x7fb5bc113810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42370/0 (socket says 172.21.15.6:42370) 2024-02-09T21:49:44.265 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.636+0000 7fb5bbfff700 1 -- 172.21.15.6:0/2685719780 learned_addr learned my addr 172.21.15.6:0/2685719780 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:49:44.265 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.636+0000 7fb5bbfff700 1 -- 172.21.15.6:0/2685719780 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ac0094f0 con 0x7fb5bc078d20 2024-02-09T21:49:44.265 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.636+0000 7fb5bbfff700 1 --2- 172.21.15.6:0/2685719780 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb5bc078d20 0x7fb5bc113810 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac00b440 tx=0x7fb5ac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:49:44.265 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.636+0000 7fb5b8ff9700 1 -- 172.21.15.6:0/2685719780 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5ac010760 con 0x7fb5bc078d20 2024-02-09T21:49:44.265 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.636+0000 7fb5c26eb700 1 -- 172.21.15.6:0/2685719780 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5bc113f50 con 0x7fb5bc078d20 2024-02-09T21:49:44.265 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.636+0000 7fb5b8ff9700 1 -- 172.21.15.6:0/2685719780 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb5ac010d50 con 0x7fb5bc078d20 2024-02-09T21:49:44.265 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.636+0000 7fb5b8ff9700 1 -- 172.21.15.6:0/2685719780 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5ac01adc0 con 0x7fb5bc078d20 2024-02-09T21:49:44.265 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.636+0000 7fb5c26eb700 1 -- 172.21.15.6:0/2685719780 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5bc114bf0 con 0x7fb5bc078d20 2024-02-09T21:49:44.265 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.637+0000 7fb5b8ff9700 1 -- 172.21.15.6:0/2685719780 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fb5ac019690 con 0x7fb5bc078d20 2024-02-09T21:49:44.265 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.637+0000 7fb5b8ff9700 1 -- 172.21.15.6:0/2685719780 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fb5ac024900 con 0x7fb5bc078d20 2024-02-09T21:49:44.265 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.637+0000 7fb5c26eb700 1 -- 172.21.15.6:0/2685719780 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5bc05f5c0 con 0x7fb5bc078d20 2024-02-09T21:49:44.265 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.640+0000 7fb5b8ff9700 1 -- 172.21.15.6:0/2685719780 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fb5ac029060 con 0x7fb5bc078d20 2024-02-09T21:49:44.265 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.709+0000 7fb5c26eb700 1 -- 172.21.15.6:0/2685719780 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fb5bc04bed0 con 0x7fb5bc078d20 2024-02-09T21:49:44.265 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.709+0000 7fb5b8ff9700 1 -- 172.21.15.6:0/2685719780 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+173 (secure 0 0 0) 0x7fb5ac01a5f0 con 0x7fb5bc078d20 2024-02-09T21:49:44.265 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.710+0000 7fb5c26eb700 1 -- 172.21.15.6:0/2685719780 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb5bc078d20 msgr2=0x7fb5bc113810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:49:44.266 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.710+0000 7fb5c26eb700 1 --2- 172.21.15.6:0/2685719780 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb5bc078d20 0x7fb5bc113810 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac00b440 tx=0x7fb5ac009250 comp rx=0 tx=0).stop 2024-02-09T21:49:44.266 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.710+0000 7fb5c26eb700 1 -- 172.21.15.6:0/2685719780 shutdown_connections 2024-02-09T21:49:44.266 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.710+0000 7fb5c26eb700 1 --2- 172.21.15.6:0/2685719780 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb5bc078d20 0x7fb5bc113810 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:49:44.266 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.710+0000 7fb5c26eb700 1 -- 172.21.15.6:0/2685719780 >> 172.21.15.6:0/2685719780 conn(0x7fb5bc0fded0 msgr2=0x7fb5bc0ff5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:49:44.266 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.710+0000 7fb5c26eb700 1 -- 172.21.15.6:0/2685719780 shutdown_connections 2024-02-09T21:49:44.266 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:43.710+0000 7fb5c26eb700 1 -- 172.21.15.6:0/2685719780 wait complete. 2024-02-09T21:49:44.266 INFO:teuthology.orchestra.run.smithi006.stdout:Restarting the monitor... 2024-02-09T21:49:44.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:44 smithi006 systemd[1]: Stopping Ceph mon.smithi006 for ee26d820-c794-11ee-95b8-87774f69a715... 2024-02-09T21:49:45.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[28880]: 2024-02-09T21:49:45.067+0000 7f9cc3868700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi006 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T21:49:45.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[28880]: 2024-02-09T21:49:45.067+0000 7f9cc3868700 -1 mon.smithi006@0(leader) e1 *** Got Signal Terminated *** 2024-02-09T21:49:45.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:45 smithi006 podman[29131]: 2024-02-09 21:49:45.181241525 +0000 UTC m=+0.760094576 container died 7ebe0e8e8eaac94c06d9291d9d93ad0fd3d788e8f094e88cc3f1836d2c7d3390 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=quincy-72ce7c1, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-02-09T21:49:45.851 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:45 smithi006 podman[29131]: 2024-02-09 21:49:45.492133452 +0000 UTC m=+1.070986500 container cleanup 7ebe0e8e8eaac94c06d9291d9d93ad0fd3d788e8f094e88cc3f1836d2c7d3390 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-72ce7c1) 2024-02-09T21:49:45.851 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:45 smithi006 bash[29131]: ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006 2024-02-09T21:49:45.851 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:45 smithi006 podman[29145]: 2024-02-09 21:49:45.643098869 +0000 UTC m=+0.454283628 container remove 7ebe0e8e8eaac94c06d9291d9d93ad0fd3d788e8f094e88cc3f1836d2c7d3390 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-72ce7c1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, ceph=True) 2024-02-09T21:49:46.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:45 smithi006 systemd[1]: ceph-ee26d820-c794-11ee-95b8-87774f69a715@mon.smithi006.service: Deactivated successfully. 2024-02-09T21:49:46.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:45 smithi006 systemd[1]: Stopped Ceph mon.smithi006 for ee26d820-c794-11ee-95b8-87774f69a715. 2024-02-09T21:49:46.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:45 smithi006 systemd[1]: Starting Ceph mon.smithi006 for ee26d820-c794-11ee-95b8-87774f69a715... 2024-02-09T21:49:46.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 podman[29240]: 2024-02-09 21:49:46.077248065 +0000 UTC m=+0.110934203 container create f8cba7495eab196f0f556df5598c0e7b667a146011c07d6ae2ff27f1a7b78f0d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006, maintainer=Guillaume Abrioux , RELEASE=quincy-72ce7c1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True) 2024-02-09T21:49:46.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 podman[29240]: 2024-02-09 21:49:45.983784061 +0000 UTC m=+0.017470216 image pull a2a0bd126c96301261ac6c5708253ef5b41bd73b23624c6dbd2eed86fc57c9b5 quay.ceph.io/ceph-ci/ceph:quincy 2024-02-09T21:49:46.327 INFO:teuthology.orchestra.run.smithi006.stdout:Setting mon public_network to 172.21.0.0/20 2024-02-09T21:49:46.452 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 podman[29240]: 2024-02-09 21:49:46.254478147 +0000 UTC m=+0.288164299 container init f8cba7495eab196f0f556df5598c0e7b667a146011c07d6ae2ff27f1a7b78f0d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, RELEASE=quincy-72ce7c1, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-09T21:49:46.452 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 podman[29240]: 2024-02-09 21:49:46.257642797 +0000 UTC m=+0.291328937 container start f8cba7495eab196f0f556df5598c0e7b667a146011c07d6ae2ff27f1a7b78f0d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , RELEASE=quincy-72ce7c1, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-02-09T21:49:46.452 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: set uid:gid to 167:167 (ceph:ceph) 2024-02-09T21:49:46.452 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable), process ceph-mon, pid 2 2024-02-09T21:49:46.452 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: pidfile_write: ignore empty --pid-file 2024-02-09T21:49:46.452 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: load: jerasure load: lrc 2024-02-09T21:49:46.452 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: RocksDB version: 6.15.5 2024-02-09T21:49:46.452 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-09T21:49:46.452 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Compile date Feb 9 2024 2024-02-09T21:49:46.452 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: DB SUMMARY 2024-02-09T21:49:46.452 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: DB Session ID: VT9F6P6PWGP79T5QD7LU 2024-02-09T21:49:46.453 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: CURRENT file: CURRENT 2024-02-09T21:49:46.453 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: IDENTITY file: IDENTITY 2024-02-09T21:49:46.453 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-02-09T21:49:46.453 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi006/store.db dir, Total Num: 1, files: 000008.sst 2024-02-09T21:49:46.453 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi006/store.db: 000010.log size: 97753 ; 2024-02-09T21:49:46.453 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.error_if_exists: 0 2024-02-09T21:49:46.453 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.create_if_missing: 0 2024-02-09T21:49:46.453 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.paranoid_checks: 1 2024-02-09T21:49:46.453 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-09T21:49:46.453 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.env: 0x5607c869aa80 2024-02-09T21:49:46.453 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.fs: Posix File System 2024-02-09T21:49:46.453 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.info_log: 0x5607ca7adcc0 2024-02-09T21:49:46.453 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.max_file_opening_threads: 16 2024-02-09T21:49:46.453 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.statistics: (nil) 2024-02-09T21:49:46.453 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.use_fsync: 0 2024-02-09T21:49:46.454 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.max_log_file_size: 0 2024-02-09T21:49:46.455 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-09T21:49:46.455 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-09T21:49:46.455 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.keep_log_file_num: 1000 2024-02-09T21:49:46.455 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.recycle_log_file_num: 0 2024-02-09T21:49:46.455 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.allow_fallocate: 1 2024-02-09T21:49:46.455 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.allow_mmap_reads: 0 2024-02-09T21:49:46.455 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.allow_mmap_writes: 0 2024-02-09T21:49:46.455 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.use_direct_reads: 0 2024-02-09T21:49:46.456 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-09T21:49:46.456 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.create_missing_column_families: 0 2024-02-09T21:49:46.456 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.db_log_dir: 2024-02-09T21:49:46.456 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi006/store.db 2024-02-09T21:49:46.456 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-09T21:49:46.456 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-09T21:49:46.456 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-09T21:49:46.456 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-09T21:49:46.456 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-09T21:49:46.456 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-09T21:49:46.456 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.advise_random_on_open: 1 2024-02-09T21:49:46.456 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.db_write_buffer_size: 0 2024-02-09T21:49:46.456 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.write_buffer_manager: 0x5607cb522b40 2024-02-09T21:49:46.456 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-09T21:49:46.456 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-09T21:49:46.457 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-09T21:49:46.457 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-09T21:49:46.457 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.rate_limiter: (nil) 2024-02-09T21:49:46.457 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-09T21:49:46.457 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.wal_recovery_mode: 2 2024-02-09T21:49:46.457 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.enable_thread_tracking: 0 2024-02-09T21:49:46.457 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.enable_pipelined_write: 0 2024-02-09T21:49:46.457 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.unordered_write: 0 2024-02-09T21:49:46.457 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-09T21:49:46.457 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-09T21:49:46.457 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-09T21:49:46.457 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-09T21:49:46.457 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.row_cache: None 2024-02-09T21:49:46.457 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.wal_filter: None 2024-02-09T21:49:46.457 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-09T21:49:46.458 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.allow_ingest_behind: 0 2024-02-09T21:49:46.458 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.preserve_deletes: 0 2024-02-09T21:49:46.458 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.two_write_queues: 0 2024-02-09T21:49:46.458 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.manual_wal_flush: 0 2024-02-09T21:49:46.458 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.atomic_flush: 0 2024-02-09T21:49:46.458 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-09T21:49:46.458 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-09T21:49:46.458 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-09T21:49:46.458 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.log_readahead_size: 0 2024-02-09T21:49:46.458 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-09T21:49:46.458 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.best_efforts_recovery: 0 2024-02-09T21:49:46.458 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-09T21:49:46.458 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-09T21:49:46.458 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.allow_data_in_errors: 0 2024-02-09T21:49:46.458 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.db_host_id: __hostname__ 2024-02-09T21:49:46.459 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.max_background_jobs: 2 2024-02-09T21:49:46.459 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.max_background_compactions: -1 2024-02-09T21:49:46.459 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.max_subcompactions: 1 2024-02-09T21:49:46.459 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-09T21:49:46.459 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-09T21:49:46.459 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-09T21:49:46.459 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.max_total_wal_size: 0 2024-02-09T21:49:46.459 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-09T21:49:46.459 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-09T21:49:46.459 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-09T21:49:46.459 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-09T21:49:46.459 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.max_open_files: -1 2024-02-09T21:49:46.459 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.bytes_per_sync: 0 2024-02-09T21:49:46.459 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-09T21:49:46.459 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-09T21:49:46.459 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.compaction_readahead_size: 0 2024-02-09T21:49:46.460 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.max_background_flushes: -1 2024-02-09T21:49:46.460 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Compression algorithms supported: 2024-02-09T21:49:46.460 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-09T21:49:46.460 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: kZSTD supported: 0 2024-02-09T21:49:46.460 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: kXpressCompression supported: 0 2024-02-09T21:49:46.460 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: kLZ4HCCompression supported: 1 2024-02-09T21:49:46.460 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: kZlibCompression supported: 1 2024-02-09T21:49:46.460 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: kSnappyCompression supported: 1 2024-02-09T21:49:46.460 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: kLZ4Compression supported: 1 2024-02-09T21:49:46.460 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: kBZip2Compression supported: 0 2024-02-09T21:49:46.460 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-09T21:49:46.460 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi006/store.db/MANIFEST-000009 2024-02-09T21:49:46.460 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-02-09T21:49:46.460 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-09T21:49:46.460 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.merge_operator: 2024-02-09T21:49:46.461 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.compaction_filter: None 2024-02-09T21:49:46.461 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.compaction_filter_factory: None 2024-02-09T21:49:46.461 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.sst_partitioner_factory: None 2024-02-09T21:49:46.461 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-09T21:49:46.461 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-09T21:49:46.461 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5607ca802f58) 2024-02-09T21:49:46.461 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: cache_index_and_filter_blocks: 1 2024-02-09T21:49:46.461 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-09T21:49:46.461 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-09T21:49:46.461 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: pin_top_level_index_and_filter: 1 2024-02-09T21:49:46.461 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: index_type: 0 2024-02-09T21:49:46.461 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: data_block_index_type: 0 2024-02-09T21:49:46.461 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: index_shortening: 1 2024-02-09T21:49:46.461 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-09T21:49:46.461 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: hash_index_allow_collision: 1 2024-02-09T21:49:46.462 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: checksum: 1 2024-02-09T21:49:46.462 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: no_block_cache: 0 2024-02-09T21:49:46.462 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: block_cache: 0x5607ca835090 2024-02-09T21:49:46.462 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: block_cache_name: BinnedLRUCache 2024-02-09T21:49:46.462 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: block_cache_options: 2024-02-09T21:49:46.462 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: capacity : 536870912 2024-02-09T21:49:46.462 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: num_shard_bits : 4 2024-02-09T21:49:46.462 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: strict_capacity_limit : 0 2024-02-09T21:49:46.462 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: high_pri_pool_ratio: 0.000 2024-02-09T21:49:46.462 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: block_cache_compressed: (nil) 2024-02-09T21:49:46.462 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: persistent_cache: (nil) 2024-02-09T21:49:46.462 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: block_size: 4096 2024-02-09T21:49:46.462 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: block_size_deviation: 10 2024-02-09T21:49:46.462 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: block_restart_interval: 16 2024-02-09T21:49:46.462 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: index_block_restart_interval: 1 2024-02-09T21:49:46.462 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: metadata_block_size: 4096 2024-02-09T21:49:46.463 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: partition_filters: 0 2024-02-09T21:49:46.463 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: use_delta_encoding: 1 2024-02-09T21:49:46.463 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-09T21:49:46.463 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: whole_key_filtering: 1 2024-02-09T21:49:46.463 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: verify_compression: 0 2024-02-09T21:49:46.463 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: read_amp_bytes_per_bit: 0 2024-02-09T21:49:46.463 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: format_version: 4 2024-02-09T21:49:46.463 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: enable_index_compression: 1 2024-02-09T21:49:46.463 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: block_align: 0 2024-02-09T21:49:46.463 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.write_buffer_size: 33554432 2024-02-09T21:49:46.463 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.max_write_buffer_number: 2 2024-02-09T21:49:46.463 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.compression: NoCompression 2024-02-09T21:49:46.463 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.bottommost_compression: Disabled 2024-02-09T21:49:46.463 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.prefix_extractor: nullptr 2024-02-09T21:49:46.463 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-09T21:49:46.463 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.num_levels: 7 2024-02-09T21:49:46.463 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-09T21:49:46.464 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-09T21:49:46.464 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-09T21:49:46.464 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-09T21:49:46.464 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-09T21:49:46.464 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-09T21:49:46.464 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-09T21:49:46.464 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-09T21:49:46.464 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-09T21:49:46.464 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-09T21:49:46.464 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-09T21:49:46.464 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.compression_opts.level: 32767 2024-02-09T21:49:46.464 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.compression_opts.strategy: 0 2024-02-09T21:49:46.464 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-09T21:49:46.464 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-09T21:49:46.464 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-09T21:49:46.465 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.compression_opts.enabled: false 2024-02-09T21:49:46.465 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-09T21:49:46.465 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-09T21:49:46.465 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-09T21:49:46.465 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.target_file_size_base: 67108864 2024-02-09T21:49:46.465 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-09T21:49:46.465 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-09T21:49:46.465 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-09T21:49:46.465 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-09T21:49:46.465 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-09T21:49:46.465 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-09T21:49:46.465 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-09T21:49:46.465 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-09T21:49:46.465 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-09T21:49:46.465 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-09T21:49:46.466 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-09T21:49:46.466 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-09T21:49:46.466 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-09T21:49:46.466 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.arena_block_size: 4194304 2024-02-09T21:49:46.466 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-09T21:49:46.466 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-09T21:49:46.466 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-09T21:49:46.466 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.disable_auto_compactions: 0 2024-02-09T21:49:46.466 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-09T21:49:46.466 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-09T21:49:46.466 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-09T21:49:46.466 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-09T21:49:46.466 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-09T21:49:46.466 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-09T21:49:46.466 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-09T21:49:46.466 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-09T21:49:46.467 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-09T21:49:46.467 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-09T21:49:46.467 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.table_properties_collectors: 2024-02-09T21:49:46.467 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.inplace_update_support: 0 2024-02-09T21:49:46.467 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-09T21:49:46.467 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-09T21:49:46.467 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-09T21:49:46.467 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-09T21:49:46.467 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.bloom_locality: 0 2024-02-09T21:49:46.467 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.max_successive_merges: 0 2024-02-09T21:49:46.467 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-09T21:49:46.467 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.paranoid_file_checks: 0 2024-02-09T21:49:46.467 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.force_consistency_checks: 1 2024-02-09T21:49:46.467 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.report_bg_io_stats: 0 2024-02-09T21:49:46.467 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.ttl: 2592000 2024-02-09T21:49:46.467 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-09T21:49:46.468 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.enable_blob_files: false 2024-02-09T21:49:46.468 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.min_blob_size: 0 2024-02-09T21:49:46.468 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.blob_file_size: 268435456 2024-02-09T21:49:46.468 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-09T21:49:46.468 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-09T21:49:46.468 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-09T21:49:46.468 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi006/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-09T21:49:46.468 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-02-09T21:49:46.468 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-02-09T21:49:46.468 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707515386283066, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-02-09T21:49:46.468 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-02-09T21:49:46.468 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-09T21:49:46.468 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707515386283820, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 92782, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 90877, "index_size": 265, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13372, "raw_average_key_size": 49, "raw_value_size": 83578, "raw_average_value_size": 309, "num_data_blocks": 11, "num_entries": 270, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707515386, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "0facb13a-919d-4348-a33c-5f3df6df9b16", "db_session_id": "VT9F6P6PWGP79T5QD7LU"}} 2024-02-09T21:49:46.468 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-02-09T21:49:46.468 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707515386284143, "job": 1, "event": "recovery_finished"} 2024-02-09T21:49:46.469 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi006/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-09T21:49:46.469 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x5607ca7c9180 2024-02-09T21:49:46.469 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: DB pointer 0x5607ca89e000 2024-02-09T21:49:46.469 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-02-09T21:49:46.469 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-02-09T21:49:46.469 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: ** DB Stats ** 2024-02-09T21:49:46.469 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-09T21:49:46.469 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-09T21:49:46.469 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-09T21:49:46.469 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-09T21:49:46.469 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-09T21:49:46.469 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-09T21:49:46.469 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-09T21:49:46.469 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: 2024-02-09T21:49:46.469 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: ** Compaction Stats [default] ** 2024-02-09T21:49:46.470 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-09T21:49:46.470 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-09T21:49:46.470 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: L0 2/0 92.41 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 171.8 0.00 0.00 1 0.001 0 0 2024-02-09T21:49:46.470 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: Sum 2/0 92.41 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 171.8 0.00 0.00 1 0.001 0 0 2024-02-09T21:49:46.470 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 171.8 0.00 0.00 1 0.001 0 0 2024-02-09T21:49:46.470 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: 2024-02-09T21:49:46.470 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: ** Compaction Stats [default] ** 2024-02-09T21:49:46.470 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-09T21:49:46.470 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-09T21:49:46.470 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 171.8 0.00 0.00 1 0.001 0 0 2024-02-09T21:49:46.470 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-09T21:49:46.470 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-09T21:49:46.470 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-09T21:49:46.470 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-09T21:49:46.470 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-09T21:49:46.471 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-09T21:49:46.471 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: Cumulative compaction: 0.00 GB write, 30.18 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-09T21:49:46.471 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: Interval compaction: 0.00 GB write, 30.18 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-09T21:49:46.471 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-09T21:49:46.471 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: 2024-02-09T21:49:46.471 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-09T21:49:46.471 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: 2024-02-09T21:49:46.471 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: ** Compaction Stats [default] ** 2024-02-09T21:49:46.471 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-09T21:49:46.471 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-09T21:49:46.471 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: L0 2/0 92.41 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 171.8 0.00 0.00 1 0.001 0 0 2024-02-09T21:49:46.471 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: Sum 2/0 92.41 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 171.8 0.00 0.00 1 0.001 0 0 2024-02-09T21:49:46.471 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-09T21:49:46.471 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: 2024-02-09T21:49:46.471 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: ** Compaction Stats [default] ** 2024-02-09T21:49:46.471 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-09T21:49:46.472 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-09T21:49:46.472 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 171.8 0.00 0.00 1 0.001 0 0 2024-02-09T21:49:46.472 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-09T21:49:46.472 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-09T21:49:46.472 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-09T21:49:46.472 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-09T21:49:46.472 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-09T21:49:46.472 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-09T21:49:46.472 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: Cumulative compaction: 0.00 GB write, 29.87 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-09T21:49:46.472 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-09T21:49:46.472 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-09T21:49:46.472 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: 2024-02-09T21:49:46.472 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-09T21:49:46.472 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: starting mon.smithi006 rank 0 at public addrs [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] at bind addrs [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi006 fsid ee26d820-c794-11ee-95b8-87774f69a715 2024-02-09T21:49:46.472 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: mon.smithi006@-1(???) e1 preinit fsid ee26d820-c794-11ee-95b8-87774f69a715 2024-02-09T21:49:46.473 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: mon.smithi006@-1(???).mds e1 new map 2024-02-09T21:49:46.473 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: mon.smithi006@-1(???).mds e1 print_map 2024-02-09T21:49:46.473 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: e1 2024-02-09T21:49:46.473 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-09T21:49:46.473 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-09T21:49:46.473 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: legacy client fscid: -1 2024-02-09T21:49:46.473 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: 2024-02-09T21:49:46.473 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout: No filesystems configured 2024-02-09T21:49:46.473 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: mon.smithi006@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-09T21:49:46.473 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: mon.smithi006@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-09T21:49:46.473 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: mon.smithi006@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-09T21:49:46.473 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: mon.smithi006@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-09T21:49:46.473 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: mon.smithi006@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-09T21:49:46.473 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 bash[29240]: f8cba7495eab196f0f556df5598c0e7b667a146011c07d6ae2ff27f1a7b78f0d 2024-02-09T21:49:46.473 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 systemd[1]: Started Ceph mon.smithi006 for ee26d820-c794-11ee-95b8-87774f69a715. 2024-02-09T21:49:46.473 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: mon.smithi006 is new leader, mons smithi006 in quorum (ranks 0) 2024-02-09T21:49:46.473 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: monmap e1: 1 mons at {smithi006=[v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-09T21:49:46.474 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: fsmap 2024-02-09T21:49:46.474 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: osdmap e1: 0 total, 0 up, 0 in 2024-02-09T21:49:46.474 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:46 smithi006 ceph-mon[29258]: mgrmap e1: no daemons active 2024-02-09T21:49:47.413 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.799+0000 7fb6d60bf700 1 Processor -- start 2024-02-09T21:49:47.413 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.799+0000 7fb6d60bf700 1 -- start start 2024-02-09T21:49:47.413 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.799+0000 7fb6d60bf700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6d0102770 0x7fb6d0102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:49:47.414 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.799+0000 7fb6d60bf700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d01030c0 con 0x7fb6d0102770 2024-02-09T21:49:47.414 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.799+0000 7fb6cf7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6d0102770 0x7fb6d0102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:49:47.414 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.799+0000 7fb6cf7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6d0102770 0x7fb6d0102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42382/0 (socket says 172.21.15.6:42382) 2024-02-09T21:49:47.414 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.799+0000 7fb6cf7fe700 1 -- 172.21.15.6:0/1729117761 learned_addr learned my addr 172.21.15.6:0/1729117761 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:49:47.414 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.799+0000 7fb6cf7fe700 1 -- 172.21.15.6:0/1729117761 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d0103200 con 0x7fb6d0102770 2024-02-09T21:49:47.414 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.800+0000 7fb6cf7fe700 1 --2- 172.21.15.6:0/1729117761 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6d0102770 0x7fb6d0102b80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb6c00098e0 tx=0x7fb6c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c54683da1bde05dc server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:49:47.414 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.800+0000 7fb6ce7fc700 1 -- 172.21.15.6:0/1729117761 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6c000b860 con 0x7fb6d0102770 2024-02-09T21:49:47.414 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.800+0000 7fb6ce7fc700 1 -- 172.21.15.6:0/1729117761 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb6c000be50 con 0x7fb6d0102770 2024-02-09T21:49:47.414 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.800+0000 7fb6ce7fc700 1 -- 172.21.15.6:0/1729117761 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6c0010380 con 0x7fb6d0102770 2024-02-09T21:49:47.414 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.800+0000 7fb6d60bf700 1 -- 172.21.15.6:0/1729117761 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6d0102770 msgr2=0x7fb6d0102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:49:47.414 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.800+0000 7fb6d60bf700 1 --2- 172.21.15.6:0/1729117761 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6d0102770 0x7fb6d0102b80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb6c00098e0 tx=0x7fb6c0009250 comp rx=0 tx=0).stop 2024-02-09T21:49:47.414 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.800+0000 7fb6d60bf700 1 -- 172.21.15.6:0/1729117761 shutdown_connections 2024-02-09T21:49:47.414 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.800+0000 7fb6d60bf700 1 --2- 172.21.15.6:0/1729117761 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6d0102770 0x7fb6d0102b80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:49:47.414 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.800+0000 7fb6d60bf700 1 -- 172.21.15.6:0/1729117761 >> 172.21.15.6:0/1729117761 conn(0x7fb6d00fded0 msgr2=0x7fb6d0100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:49:47.414 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.800+0000 7fb6d60bf700 1 -- 172.21.15.6:0/1729117761 shutdown_connections 2024-02-09T21:49:47.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.800+0000 7fb6d60bf700 1 -- 172.21.15.6:0/1729117761 wait complete. 2024-02-09T21:49:47.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.801+0000 7fb6d60bf700 1 Processor -- start 2024-02-09T21:49:47.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.801+0000 7fb6d60bf700 1 -- start start 2024-02-09T21:49:47.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.801+0000 7fb6d60bf700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6d0102770 0x7fb6d0111590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:49:47.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.801+0000 7fb6d60bf700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d0111ad0 con 0x7fb6d0102770 2024-02-09T21:49:47.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.801+0000 7fb6cf7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6d0102770 0x7fb6d0111590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:49:47.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.801+0000 7fb6cf7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6d0102770 0x7fb6d0111590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42392/0 (socket says 172.21.15.6:42392) 2024-02-09T21:49:47.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.801+0000 7fb6cf7fe700 1 -- 172.21.15.6:0/815888155 learned_addr learned my addr 172.21.15.6:0/815888155 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:49:47.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.802+0000 7fb6cf7fe700 1 -- 172.21.15.6:0/815888155 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c00094f0 con 0x7fb6d0102770 2024-02-09T21:49:47.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.802+0000 7fb6cf7fe700 1 --2- 172.21.15.6:0/815888155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6d0102770 0x7fb6d0111590 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb6c000b440 tx=0x7fb6c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:49:47.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.802+0000 7fb6b7fff700 1 -- 172.21.15.6:0/815888155 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6c0010760 con 0x7fb6d0102770 2024-02-09T21:49:47.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.802+0000 7fb6b7fff700 1 -- 172.21.15.6:0/815888155 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb6c0010d50 con 0x7fb6d0102770 2024-02-09T21:49:47.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.802+0000 7fb6d60bf700 1 -- 172.21.15.6:0/815888155 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6d0111cd0 con 0x7fb6d0102770 2024-02-09T21:49:47.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.802+0000 7fb6b7fff700 1 -- 172.21.15.6:0/815888155 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6c001ae20 con 0x7fb6d0102770 2024-02-09T21:49:47.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.802+0000 7fb6d60bf700 1 -- 172.21.15.6:0/815888155 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6d0112970 con 0x7fb6d0102770 2024-02-09T21:49:47.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.802+0000 7fb6b7fff700 1 -- 172.21.15.6:0/815888155 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fb6c00246d0 con 0x7fb6d0102770 2024-02-09T21:49:47.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.802+0000 7fb6b7fff700 1 -- 172.21.15.6:0/815888155 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fb6c002e9b0 con 0x7fb6d0102770 2024-02-09T21:49:47.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.803+0000 7fb6d60bf700 1 -- 172.21.15.6:0/815888155 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6d005f5c0 con 0x7fb6d0102770 2024-02-09T21:49:47.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.805+0000 7fb6b7fff700 1 -- 172.21.15.6:0/815888155 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fb6c0029070 con 0x7fb6d0102770 2024-02-09T21:49:47.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.875+0000 7fb6d60bf700 1 -- 172.21.15.6:0/815888155 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fb6d01045e0 con 0x7fb6d0102770 2024-02-09T21:49:47.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.877+0000 7fb6b7fff700 1 -- 172.21.15.6:0/815888155 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fb6c001a6a0 con 0x7fb6d0102770 2024-02-09T21:49:47.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.878+0000 7fb6d60bf700 1 -- 172.21.15.6:0/815888155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6d0102770 msgr2=0x7fb6d0111590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:49:47.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.878+0000 7fb6d60bf700 1 --2- 172.21.15.6:0/815888155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6d0102770 0x7fb6d0111590 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb6c000b440 tx=0x7fb6c0009250 comp rx=0 tx=0).stop 2024-02-09T21:49:47.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.878+0000 7fb6d60bf700 1 -- 172.21.15.6:0/815888155 shutdown_connections 2024-02-09T21:49:47.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.878+0000 7fb6d60bf700 1 --2- 172.21.15.6:0/815888155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6d0102770 0x7fb6d0111590 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:49:47.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.878+0000 7fb6d60bf700 1 -- 172.21.15.6:0/815888155 >> 172.21.15.6:0/815888155 conn(0x7fb6d00fded0 msgr2=0x7fb6d00ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:49:47.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.878+0000 7fb6d60bf700 1 -- 172.21.15.6:0/815888155 shutdown_connections 2024-02-09T21:49:47.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:46.878+0000 7fb6d60bf700 1 -- 172.21.15.6:0/815888155 wait complete. 2024-02-09T21:49:47.416 INFO:teuthology.orchestra.run.smithi006.stdout:Wrote config to /etc/ceph/ceph.conf 2024-02-09T21:49:47.417 INFO:teuthology.orchestra.run.smithi006.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-09T21:49:47.417 INFO:teuthology.orchestra.run.smithi006.stdout:Creating mgr... 2024-02-09T21:49:47.417 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying port 9283 ... 2024-02-09T21:49:47.589 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-ee26d820-c794-11ee-95b8-87774f69a715@mgr.smithi006.glirxt 2024-02-09T21:49:47.589 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Failed to reset failed state of unit ceph-ee26d820-c794-11ee-95b8-87774f69a715@mgr.smithi006.glirxt.service: Unit ceph-ee26d820-c794-11ee-95b8-87774f69a715@mgr.smithi006.glirxt.service not loaded. 2024-02-09T21:49:47.745 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-ee26d820-c794-11ee-95b8-87774f69a715.target.wants/ceph-ee26d820-c794-11ee-95b8-87774f69a715@mgr.smithi006.glirxt.service → /etc/systemd/system/ceph-ee26d820-c794-11ee-95b8-87774f69a715@.service. 2024-02-09T21:49:48.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:47 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/815888155' entity='client.admin' 2024-02-09T21:49:48.323 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-09T21:49:48.324 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout disabled 2024-02-09T21:49:48.331 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-09T21:49:48.331 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout inactive 2024-02-09T21:49:48.331 INFO:teuthology.orchestra.run.smithi006.stdout:firewalld.service is not enabled 2024-02-09T21:49:48.331 INFO:teuthology.orchestra.run.smithi006.stdout:Not possible to enable service . firewalld.service is not available 2024-02-09T21:49:48.340 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-09T21:49:48.340 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout disabled 2024-02-09T21:49:48.347 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-09T21:49:48.347 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout inactive 2024-02-09T21:49:48.347 INFO:teuthology.orchestra.run.smithi006.stdout:firewalld.service is not enabled 2024-02-09T21:49:48.347 INFO:teuthology.orchestra.run.smithi006.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-09T21:49:48.347 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for mgr to start... 2024-02-09T21:49:48.347 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for mgr... 2024-02-09T21:49:49.141 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:48 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/4024354697' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-09T21:49:49.414 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-02-09T21:49:49.414 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-02-09T21:49:49.414 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsid": "ee26d820-c794-11ee-95b8-87774f69a715", 2024-02-09T21:49:49.414 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "health": { 2024-02-09T21:49:49.414 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-09T21:49:49.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-09T21:49:49.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-09T21:49:49.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-02-09T21:49:49.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-09T21:49:49.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-09T21:49:49.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 0 2024-02-09T21:49:49.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-02-09T21:49:49.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-09T21:49:49.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "smithi006" 2024-02-09T21:49:49.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-02-09T21:49:49.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-02-09T21:49:49.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-09T21:49:49.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T21:49:49.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-02-09T21:49:49.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-09T21:49:49.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-02-09T21:49:49.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-09T21:49:49.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T21:49:49.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-09T21:49:49.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-09T21:49:49.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-09T21:49:49.424 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-09T21:49:49.424 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-09T21:49:49.424 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-09T21:49:49.424 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-02-09T21:49:49.424 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-09T21:49:49.424 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-09T21:49:49.424 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-09T21:49:49.425 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-09T21:49:49.425 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-09T21:49:49.425 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-09T21:49:49.425 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-09T21:49:49.425 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-09T21:49:49.425 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-09T21:49:49.425 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-02-09T21:49:49.425 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-09T21:49:49.425 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T21:49:49.425 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-09T21:49:49.425 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-09T21:49:49.425 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-02-09T21:49:49.425 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-09T21:49:49.426 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-09T21:49:49.426 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-09T21:49:49.426 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-09T21:49:49.426 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-09T21:49:49.426 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-09T21:49:49.426 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "restful" 2024-02-09T21:49:49.426 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-02-09T21:49:49.426 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-09T21:49:49.426 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-02-09T21:49:49.426 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-09T21:49:49.426 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T21:49:49.426 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modified": "2024-02-09T21:49:39.138636+0000", 2024-02-09T21:49:49.426 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-09T21:49:49.426 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-02-09T21:49:49.426 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-09T21:49:49.427 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-02-09T21:49:49.427 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.758+0000 7f65f82aa700 1 Processor -- start 2024-02-09T21:49:49.427 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.758+0000 7f65f82aa700 1 -- start start 2024-02-09T21:49:49.427 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.758+0000 7f65f82aa700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65f0103da0 0x7f65f01041b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:49:49.427 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.758+0000 7f65f82aa700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f01046f0 con 0x7f65f0103da0 2024-02-09T21:49:49.427 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.758+0000 7f65f6046700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65f0103da0 0x7f65f01041b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:49:49.427 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.758+0000 7f65f6046700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65f0103da0 0x7f65f01041b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60620/0 (socket says 172.21.15.6:60620) 2024-02-09T21:49:49.427 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.758+0000 7f65f6046700 1 -- 172.21.15.6:0/3893469574 learned_addr learned my addr 172.21.15.6:0/3893469574 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:49:49.427 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.759+0000 7f65f6046700 1 -- 172.21.15.6:0/3893469574 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f0104830 con 0x7f65f0103da0 2024-02-09T21:49:49.427 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.759+0000 7f65f6046700 1 --2- 172.21.15.6:0/3893469574 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65f0103da0 0x7f65f01041b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f65ec0098e0 tx=0x7f65ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c7ccc1a7886eacf server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:49:49.427 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.759+0000 7f65f5044700 1 -- 172.21.15.6:0/3893469574 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65ec00b860 con 0x7f65f0103da0 2024-02-09T21:49:49.427 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.759+0000 7f65f5044700 1 -- 172.21.15.6:0/3893469574 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f65ec00be50 con 0x7f65f0103da0 2024-02-09T21:49:49.427 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.759+0000 7f65f5044700 1 -- 172.21.15.6:0/3893469574 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65ec010380 con 0x7f65f0103da0 2024-02-09T21:49:49.427 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.759+0000 7f65f82aa700 1 -- 172.21.15.6:0/3893469574 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65f0103da0 msgr2=0x7f65f01041b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:49:49.427 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.759+0000 7f65f82aa700 1 --2- 172.21.15.6:0/3893469574 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65f0103da0 0x7f65f01041b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f65ec0098e0 tx=0x7f65ec009250 comp rx=0 tx=0).stop 2024-02-09T21:49:49.428 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.760+0000 7f65f82aa700 1 -- 172.21.15.6:0/3893469574 shutdown_connections 2024-02-09T21:49:49.428 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.760+0000 7f65f82aa700 1 --2- 172.21.15.6:0/3893469574 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65f0103da0 0x7f65f01041b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:49:49.428 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.760+0000 7f65f82aa700 1 -- 172.21.15.6:0/3893469574 >> 172.21.15.6:0/3893469574 conn(0x7f65f00fc580 msgr2=0x7f65f00fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:49:49.428 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.760+0000 7f65f82aa700 1 -- 172.21.15.6:0/3893469574 shutdown_connections 2024-02-09T21:49:49.428 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.760+0000 7f65f82aa700 1 -- 172.21.15.6:0/3893469574 wait complete. 2024-02-09T21:49:49.428 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.760+0000 7f65f82aa700 1 Processor -- start 2024-02-09T21:49:49.428 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.760+0000 7f65f82aa700 1 -- start start 2024-02-09T21:49:49.428 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.760+0000 7f65f82aa700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65f0103da0 0x7f65f0111540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:49:49.428 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.760+0000 7f65f82aa700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f0111a80 con 0x7f65f0103da0 2024-02-09T21:49:49.428 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.761+0000 7f65f6046700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65f0103da0 0x7f65f0111540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:49:49.428 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.761+0000 7f65f6046700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65f0103da0 0x7f65f0111540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60632/0 (socket says 172.21.15.6:60632) 2024-02-09T21:49:49.428 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.761+0000 7f65f6046700 1 -- 172.21.15.6:0/4024354697 learned_addr learned my addr 172.21.15.6:0/4024354697 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:49:49.428 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.761+0000 7f65f6046700 1 -- 172.21.15.6:0/4024354697 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65ec0094f0 con 0x7f65f0103da0 2024-02-09T21:49:49.428 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.761+0000 7f65f6046700 1 --2- 172.21.15.6:0/4024354697 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65f0103da0 0x7f65f0111540 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f65ec000c00 tx=0x7f65ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:49:49.428 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.761+0000 7f65e6ffd700 1 -- 172.21.15.6:0/4024354697 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65ec0107c0 con 0x7f65f0103da0 2024-02-09T21:49:49.429 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.761+0000 7f65e6ffd700 1 -- 172.21.15.6:0/4024354697 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f65ec010db0 con 0x7f65f0103da0 2024-02-09T21:49:49.429 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.761+0000 7f65f82aa700 1 -- 172.21.15.6:0/4024354697 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f0111c80 con 0x7f65f0103da0 2024-02-09T21:49:49.429 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.761+0000 7f65e6ffd700 1 -- 172.21.15.6:0/4024354697 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65ec01ad00 con 0x7f65f0103da0 2024-02-09T21:49:49.429 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.761+0000 7f65f82aa700 1 -- 172.21.15.6:0/4024354697 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f0112920 con 0x7f65f0103da0 2024-02-09T21:49:49.429 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.761+0000 7f65e6ffd700 1 -- 172.21.15.6:0/4024354697 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f65ec024480 con 0x7f65f0103da0 2024-02-09T21:49:49.429 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.761+0000 7f65e6ffd700 1 -- 172.21.15.6:0/4024354697 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f65ec019710 con 0x7f65f0103da0 2024-02-09T21:49:49.429 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.762+0000 7f65f82aa700 1 -- 172.21.15.6:0/4024354697 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65f005f5c0 con 0x7f65f0103da0 2024-02-09T21:49:49.429 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.764+0000 7f65e6ffd700 1 -- 172.21.15.6:0/4024354697 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f65ec01a530 con 0x7f65f0103da0 2024-02-09T21:49:49.429 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.837+0000 7f65f82aa700 1 -- 172.21.15.6:0/4024354697 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f65f004bed0 con 0x7f65f0103da0 2024-02-09T21:49:49.429 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.837+0000 7f65e6ffd700 1 -- 172.21.15.6:0/4024354697 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f65ec010920 con 0x7f65f0103da0 2024-02-09T21:49:49.429 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.838+0000 7f65f82aa700 1 -- 172.21.15.6:0/4024354697 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65f0103da0 msgr2=0x7f65f0111540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:49:49.429 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.838+0000 7f65f82aa700 1 --2- 172.21.15.6:0/4024354697 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65f0103da0 0x7f65f0111540 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f65ec000c00 tx=0x7f65ec009250 comp rx=0 tx=0).stop 2024-02-09T21:49:49.429 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.838+0000 7f65f82aa700 1 -- 172.21.15.6:0/4024354697 shutdown_connections 2024-02-09T21:49:49.429 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.838+0000 7f65f82aa700 1 --2- 172.21.15.6:0/4024354697 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65f0103da0 0x7f65f0111540 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:49:49.429 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.838+0000 7f65f82aa700 1 -- 172.21.15.6:0/4024354697 >> 172.21.15.6:0/4024354697 conn(0x7f65f00fc580 msgr2=0x7f65f00fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:49:49.430 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.838+0000 7f65f82aa700 1 -- 172.21.15.6:0/4024354697 shutdown_connections 2024-02-09T21:49:49.430 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:48.838+0000 7f65f82aa700 1 -- 172.21.15.6:0/4024354697 wait complete. 2024-02-09T21:49:49.430 INFO:teuthology.orchestra.run.smithi006.stdout:mgr not available, waiting (1/15)... 2024-02-09T21:49:52.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:52 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/1890677352' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-09T21:49:52.503 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-02-09T21:49:52.504 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-02-09T21:49:52.504 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsid": "ee26d820-c794-11ee-95b8-87774f69a715", 2024-02-09T21:49:52.504 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "health": { 2024-02-09T21:49:52.504 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-09T21:49:52.504 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-09T21:49:52.504 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-09T21:49:52.504 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-02-09T21:49:52.504 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-09T21:49:52.504 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-09T21:49:52.504 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 0 2024-02-09T21:49:52.504 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-02-09T21:49:52.504 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-09T21:49:52.505 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "smithi006" 2024-02-09T21:49:52.505 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-02-09T21:49:52.505 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-02-09T21:49:52.505 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-09T21:49:52.505 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T21:49:52.505 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-02-09T21:49:52.505 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-09T21:49:52.505 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-02-09T21:49:52.505 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-09T21:49:52.505 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T21:49:52.505 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-09T21:49:52.505 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-09T21:49:52.505 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-09T21:49:52.513 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-09T21:49:52.513 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-09T21:49:52.513 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-09T21:49:52.513 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-02-09T21:49:52.513 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-09T21:49:52.513 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-09T21:49:52.513 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-09T21:49:52.513 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-09T21:49:52.513 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-09T21:49:52.513 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-09T21:49:52.514 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-09T21:49:52.514 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-09T21:49:52.514 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-09T21:49:52.514 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-02-09T21:49:52.514 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-09T21:49:52.514 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T21:49:52.514 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-09T21:49:52.514 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-09T21:49:52.514 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-02-09T21:49:52.514 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-09T21:49:52.514 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-09T21:49:52.514 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-09T21:49:52.514 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-09T21:49:52.514 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-09T21:49:52.514 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-09T21:49:52.514 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "restful" 2024-02-09T21:49:52.515 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-02-09T21:49:52.515 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-09T21:49:52.515 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-02-09T21:49:52.515 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-09T21:49:52.515 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T21:49:52.515 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modified": "2024-02-09T21:49:39.138636+0000", 2024-02-09T21:49:52.515 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-09T21:49:52.515 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-02-09T21:49:52.515 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-09T21:49:52.515 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-02-09T21:49:52.515 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.771+0000 7f5245f7d700 1 Processor -- start 2024-02-09T21:49:52.515 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.771+0000 7f5245f7d700 1 -- start start 2024-02-09T21:49:52.515 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.771+0000 7f5245f7d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5240076c60 0x7f5240105340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:49:52.515 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.771+0000 7f5245f7d700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5240105880 con 0x7f5240076c60 2024-02-09T21:49:52.515 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.772+0000 7f523f7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5240076c60 0x7f5240105340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:49:52.515 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.772+0000 7f523f7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5240076c60 0x7f5240105340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60636/0 (socket says 172.21.15.6:60636) 2024-02-09T21:49:52.516 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.772+0000 7f523f7fe700 1 -- 172.21.15.6:0/3971684521 learned_addr learned my addr 172.21.15.6:0/3971684521 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:49:52.516 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.772+0000 7f523f7fe700 1 -- 172.21.15.6:0/3971684521 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52401059c0 con 0x7f5240076c60 2024-02-09T21:49:52.516 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.772+0000 7f523f7fe700 1 --2- 172.21.15.6:0/3971684521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5240076c60 0x7f5240105340 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f52280098e0 tx=0x7f5228009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21bd228a09593fcc server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:49:52.516 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.772+0000 7f523e7fc700 1 -- 172.21.15.6:0/3971684521 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f522800b860 con 0x7f5240076c60 2024-02-09T21:49:52.516 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.772+0000 7f523e7fc700 1 -- 172.21.15.6:0/3971684521 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f522800be50 con 0x7f5240076c60 2024-02-09T21:49:52.516 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.772+0000 7f523e7fc700 1 -- 172.21.15.6:0/3971684521 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5228010380 con 0x7f5240076c60 2024-02-09T21:49:52.516 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.773+0000 7f5245f7d700 1 -- 172.21.15.6:0/3971684521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5240076c60 msgr2=0x7f5240105340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:49:52.516 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.773+0000 7f5245f7d700 1 --2- 172.21.15.6:0/3971684521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5240076c60 0x7f5240105340 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f52280098e0 tx=0x7f5228009250 comp rx=0 tx=0).stop 2024-02-09T21:49:52.519 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.773+0000 7f5245f7d700 1 -- 172.21.15.6:0/3971684521 shutdown_connections 2024-02-09T21:49:52.519 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.773+0000 7f5245f7d700 1 --2- 172.21.15.6:0/3971684521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5240076c60 0x7f5240105340 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:49:52.519 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.773+0000 7f5245f7d700 1 -- 172.21.15.6:0/3971684521 >> 172.21.15.6:0/3971684521 conn(0x7f52400fcdd0 msgr2=0x7f52400ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:49:52.519 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.773+0000 7f5245f7d700 1 -- 172.21.15.6:0/3971684521 shutdown_connections 2024-02-09T21:49:52.519 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.773+0000 7f5245f7d700 1 -- 172.21.15.6:0/3971684521 wait complete. 2024-02-09T21:49:52.519 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.773+0000 7f5245f7d700 1 Processor -- start 2024-02-09T21:49:52.519 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.774+0000 7f5245f7d700 1 -- start start 2024-02-09T21:49:52.519 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.774+0000 7f5245f7d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5240076c60 0x7f5240111520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:49:52.519 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.774+0000 7f5245f7d700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5240111a60 con 0x7f5240076c60 2024-02-09T21:49:52.520 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.774+0000 7f523f7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5240076c60 0x7f5240111520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:49:52.520 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.774+0000 7f523f7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5240076c60 0x7f5240111520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60642/0 (socket says 172.21.15.6:60642) 2024-02-09T21:49:52.520 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.774+0000 7f523f7fe700 1 -- 172.21.15.6:0/1890677352 learned_addr learned my addr 172.21.15.6:0/1890677352 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:49:52.520 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.774+0000 7f523f7fe700 1 -- 172.21.15.6:0/1890677352 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52280094f0 con 0x7f5240076c60 2024-02-09T21:49:52.520 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.774+0000 7f523f7fe700 1 --2- 172.21.15.6:0/1890677352 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5240076c60 0x7f5240111520 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5228006d40 tx=0x7f5228009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:49:52.520 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.775+0000 7f5237fff700 1 -- 172.21.15.6:0/1890677352 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5228010760 con 0x7f5240076c60 2024-02-09T21:49:52.520 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.775+0000 7f5237fff700 1 -- 172.21.15.6:0/1890677352 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f5228010d50 con 0x7f5240076c60 2024-02-09T21:49:52.520 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.775+0000 7f5237fff700 1 -- 172.21.15.6:0/1890677352 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5228024d20 con 0x7f5240076c60 2024-02-09T21:49:52.520 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.775+0000 7f5245f7d700 1 -- 172.21.15.6:0/1890677352 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5240111c60 con 0x7f5240076c60 2024-02-09T21:49:52.520 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.775+0000 7f5245f7d700 1 -- 172.21.15.6:0/1890677352 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5240112900 con 0x7f5240076c60 2024-02-09T21:49:52.520 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.775+0000 7f5237fff700 1 -- 172.21.15.6:0/1890677352 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f522801a6b0 con 0x7f5240076c60 2024-02-09T21:49:52.520 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.775+0000 7f5237fff700 1 -- 172.21.15.6:0/1890677352 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f52280199d0 con 0x7f5240076c60 2024-02-09T21:49:52.520 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.776+0000 7f5245f7d700 1 -- 172.21.15.6:0/1890677352 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f524005f5c0 con 0x7f5240076c60 2024-02-09T21:49:52.520 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.778+0000 7f5237fff700 1 -- 172.21.15.6:0/1890677352 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f5228029070 con 0x7f5240076c60 2024-02-09T21:49:52.521 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.852+0000 7f5245f7d700 1 -- 172.21.15.6:0/1890677352 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f524004bed0 con 0x7f5240076c60 2024-02-09T21:49:52.521 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.853+0000 7f5237fff700 1 -- 172.21.15.6:0/1890677352 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f5228019b90 con 0x7f5240076c60 2024-02-09T21:49:52.521 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.854+0000 7f5245f7d700 1 -- 172.21.15.6:0/1890677352 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5240076c60 msgr2=0x7f5240111520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:49:52.521 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.854+0000 7f5245f7d700 1 --2- 172.21.15.6:0/1890677352 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5240076c60 0x7f5240111520 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5228006d40 tx=0x7f5228009250 comp rx=0 tx=0).stop 2024-02-09T21:49:52.521 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.854+0000 7f5245f7d700 1 -- 172.21.15.6:0/1890677352 shutdown_connections 2024-02-09T21:49:52.521 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.854+0000 7f5245f7d700 1 --2- 172.21.15.6:0/1890677352 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5240076c60 0x7f5240111520 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:49:52.521 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.854+0000 7f5245f7d700 1 -- 172.21.15.6:0/1890677352 >> 172.21.15.6:0/1890677352 conn(0x7f52400fcdd0 msgr2=0x7f52400fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:49:52.521 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.854+0000 7f5245f7d700 1 -- 172.21.15.6:0/1890677352 shutdown_connections 2024-02-09T21:49:52.521 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:51.854+0000 7f5245f7d700 1 -- 172.21.15.6:0/1890677352 wait complete. 2024-02-09T21:49:52.521 INFO:teuthology.orchestra.run.smithi006.stdout:mgr not available, waiting (2/15)... 2024-02-09T21:49:55.297 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:54 smithi006 ceph-mon[29258]: Activating manager daemon smithi006.glirxt 2024-02-09T21:49:55.297 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:54 smithi006 ceph-mon[29258]: mgrmap e2: smithi006.glirxt(active, starting, since 0.00300603s) 2024-02-09T21:49:55.297 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:54 smithi006 ceph-mon[29258]: from='mgr.14100 172.21.15.6:0/3743359400' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-09T21:49:55.297 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:54 smithi006 ceph-mon[29258]: from='mgr.14100 172.21.15.6:0/3743359400' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-09T21:49:55.298 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:54 smithi006 ceph-mon[29258]: from='mgr.14100 172.21.15.6:0/3743359400' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-09T21:49:55.298 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:54 smithi006 ceph-mon[29258]: from='mgr.14100 172.21.15.6:0/3743359400' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata", "id": "smithi006"}]: dispatch 2024-02-09T21:49:55.298 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:54 smithi006 ceph-mon[29258]: from='mgr.14100 172.21.15.6:0/3743359400' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mgr metadata", "who": "smithi006.glirxt", "id": "smithi006.glirxt"}]: dispatch 2024-02-09T21:49:55.298 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:54 smithi006 ceph-mon[29258]: Manager daemon smithi006.glirxt is now available 2024-02-09T21:49:55.298 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:54 smithi006 ceph-mon[29258]: from='mgr.14100 172.21.15.6:0/3743359400' entity='mgr.smithi006.glirxt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi006.glirxt/mirror_snapshot_schedule"}]: dispatch 2024-02-09T21:49:55.298 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:54 smithi006 ceph-mon[29258]: from='mgr.14100 172.21.15.6:0/3743359400' entity='mgr.smithi006.glirxt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi006.glirxt/trash_purge_schedule"}]: dispatch 2024-02-09T21:49:55.298 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:54 smithi006 ceph-mon[29258]: from='mgr.14100 172.21.15.6:0/3743359400' entity='mgr.smithi006.glirxt' 2024-02-09T21:49:55.298 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:54 smithi006 ceph-mon[29258]: from='mgr.14100 172.21.15.6:0/3743359400' entity='mgr.smithi006.glirxt' 2024-02-09T21:49:55.619 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-02-09T21:49:55.619 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-02-09T21:49:55.619 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsid": "ee26d820-c794-11ee-95b8-87774f69a715", 2024-02-09T21:49:55.619 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "health": { 2024-02-09T21:49:55.620 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-09T21:49:55.620 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-09T21:49:55.620 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-09T21:49:55.620 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-02-09T21:49:55.620 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-09T21:49:55.620 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-09T21:49:55.620 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 0 2024-02-09T21:49:55.620 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-02-09T21:49:55.620 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-09T21:49:55.620 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "smithi006" 2024-02-09T21:49:55.620 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-02-09T21:49:55.620 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-02-09T21:49:55.620 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-09T21:49:55.620 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T21:49:55.620 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-02-09T21:49:55.620 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-09T21:49:55.621 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-02-09T21:49:55.621 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-09T21:49:55.621 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T21:49:55.621 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-09T21:49:55.621 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-09T21:49:55.621 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-09T21:49:55.628 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-09T21:49:55.629 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-09T21:49:55.629 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-09T21:49:55.629 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-02-09T21:49:55.629 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-09T21:49:55.629 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-09T21:49:55.629 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-09T21:49:55.629 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-09T21:49:55.629 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-09T21:49:55.629 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-09T21:49:55.629 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-09T21:49:55.629 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-09T21:49:55.629 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-09T21:49:55.629 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-02-09T21:49:55.629 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-09T21:49:55.630 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T21:49:55.630 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-09T21:49:55.630 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-09T21:49:55.630 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-02-09T21:49:55.630 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-09T21:49:55.630 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-09T21:49:55.630 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-09T21:49:55.630 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-09T21:49:55.630 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-09T21:49:55.630 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-09T21:49:55.630 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "restful" 2024-02-09T21:49:55.630 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-02-09T21:49:55.630 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-09T21:49:55.630 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-02-09T21:49:55.630 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-09T21:49:55.630 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T21:49:55.631 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modified": "2024-02-09T21:49:39.138636+0000", 2024-02-09T21:49:55.631 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-09T21:49:55.631 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-02-09T21:49:55.631 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-09T21:49:55.631 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-02-09T21:49:55.631 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.914+0000 7fae3c977700 1 Processor -- start 2024-02-09T21:49:55.631 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.914+0000 7fae3c977700 1 -- start start 2024-02-09T21:49:55.631 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.915+0000 7fae3c977700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae38102c60 0x7fae38103070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:49:55.631 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.915+0000 7fae3c977700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae381035b0 con 0x7fae38102c60 2024-02-09T21:49:55.631 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.915+0000 7fae3659c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae38102c60 0x7fae38103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:49:55.631 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.915+0000 7fae3659c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae38102c60 0x7fae38103070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60718/0 (socket says 172.21.15.6:60718) 2024-02-09T21:49:55.631 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.915+0000 7fae3659c700 1 -- 172.21.15.6:0/1646187670 learned_addr learned my addr 172.21.15.6:0/1646187670 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:49:55.631 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.915+0000 7fae3659c700 1 -- 172.21.15.6:0/1646187670 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae381036f0 con 0x7fae38102c60 2024-02-09T21:49:55.631 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.916+0000 7fae3659c700 1 --2- 172.21.15.6:0/1646187670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae38102c60 0x7fae38103070 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fae280098e0 tx=0x7fae28009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae9a589d73f8f651 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:49:55.631 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.916+0000 7fae3559a700 1 -- 172.21.15.6:0/1646187670 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae2800b860 con 0x7fae38102c60 2024-02-09T21:49:55.631 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.916+0000 7fae3559a700 1 -- 172.21.15.6:0/1646187670 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fae2800be50 con 0x7fae38102c60 2024-02-09T21:49:55.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.916+0000 7fae3559a700 1 -- 172.21.15.6:0/1646187670 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae28018420 con 0x7fae38102c60 2024-02-09T21:49:55.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.917+0000 7fae3c977700 1 -- 172.21.15.6:0/1646187670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae38102c60 msgr2=0x7fae38103070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:49:55.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.917+0000 7fae3c977700 1 --2- 172.21.15.6:0/1646187670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae38102c60 0x7fae38103070 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fae280098e0 tx=0x7fae28009250 comp rx=0 tx=0).stop 2024-02-09T21:49:55.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.917+0000 7fae3c977700 1 -- 172.21.15.6:0/1646187670 shutdown_connections 2024-02-09T21:49:55.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.917+0000 7fae3c977700 1 --2- 172.21.15.6:0/1646187670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae38102c60 0x7fae38103070 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:49:55.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.917+0000 7fae3c977700 1 -- 172.21.15.6:0/1646187670 >> 172.21.15.6:0/1646187670 conn(0x7fae380fc580 msgr2=0x7fae380fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:49:55.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.917+0000 7fae3c977700 1 -- 172.21.15.6:0/1646187670 shutdown_connections 2024-02-09T21:49:55.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.917+0000 7fae3c977700 1 -- 172.21.15.6:0/1646187670 wait complete. 2024-02-09T21:49:55.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.918+0000 7fae3c977700 1 Processor -- start 2024-02-09T21:49:55.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.918+0000 7fae3c977700 1 -- start start 2024-02-09T21:49:55.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.918+0000 7fae3c977700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae38102c60 0x7fae38113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:49:55.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.918+0000 7fae3c977700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae38113ec0 con 0x7fae38102c60 2024-02-09T21:49:55.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.918+0000 7fae3659c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae38102c60 0x7fae38113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:49:55.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.918+0000 7fae3659c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae38102c60 0x7fae38113980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60724/0 (socket says 172.21.15.6:60724) 2024-02-09T21:49:55.632 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.918+0000 7fae3659c700 1 -- 172.21.15.6:0/441021261 learned_addr learned my addr 172.21.15.6:0/441021261 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:49:55.633 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.918+0000 7fae3659c700 1 -- 172.21.15.6:0/441021261 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae280094f0 con 0x7fae38102c60 2024-02-09T21:49:55.633 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.918+0000 7fae3659c700 1 --2- 172.21.15.6:0/441021261 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae38102c60 0x7fae38113980 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fae28000c00 tx=0x7fae28009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:49:55.633 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.919+0000 7fae26ffd700 1 -- 172.21.15.6:0/441021261 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae28018860 con 0x7fae38102c60 2024-02-09T21:49:55.633 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.919+0000 7fae26ffd700 1 -- 172.21.15.6:0/441021261 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fae28018e50 con 0x7fae38102c60 2024-02-09T21:49:55.633 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.919+0000 7fae26ffd700 1 -- 172.21.15.6:0/441021261 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae28021390 con 0x7fae38102c60 2024-02-09T21:49:55.633 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.919+0000 7fae3c977700 1 -- 172.21.15.6:0/441021261 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae381140c0 con 0x7fae38102c60 2024-02-09T21:49:55.633 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.919+0000 7fae3c977700 1 -- 172.21.15.6:0/441021261 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae38114d60 con 0x7fae38102c60 2024-02-09T21:49:55.633 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.920+0000 7fae3c977700 1 -- 172.21.15.6:0/441021261 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae3805f5c0 con 0x7fae38102c60 2024-02-09T21:49:55.633 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.921+0000 7fae26ffd700 1 -- 172.21.15.6:0/441021261 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 2) v1 ==== 43124+0+0 (secure 0 0 0) 0x7fae28022430 con 0x7fae38102c60 2024-02-09T21:49:55.633 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.921+0000 7fae26ffd700 1 -- 172.21.15.6:0/441021261 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fae28014070 con 0x7fae38102c60 2024-02-09T21:49:55.633 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.923+0000 7fae26ffd700 1 -- 172.21.15.6:0/441021261 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fae2800fc50 con 0x7fae38102c60 2024-02-09T21:49:55.633 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.998+0000 7fae3c977700 1 -- 172.21.15.6:0/441021261 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fae38101520 con 0x7fae38102c60 2024-02-09T21:49:55.633 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.999+0000 7fae26ffd700 1 -- 172.21.15.6:0/441021261 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fae28027070 con 0x7fae38102c60 2024-02-09T21:49:55.633 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.999+0000 7fae3c977700 1 -- 172.21.15.6:0/441021261 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae38102c60 msgr2=0x7fae38113980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:49:55.633 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.999+0000 7fae3c977700 1 --2- 172.21.15.6:0/441021261 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae38102c60 0x7fae38113980 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fae28000c00 tx=0x7fae28009250 comp rx=0 tx=0).stop 2024-02-09T21:49:55.633 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.999+0000 7fae3c977700 1 -- 172.21.15.6:0/441021261 shutdown_connections 2024-02-09T21:49:55.633 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.999+0000 7fae3c977700 1 --2- 172.21.15.6:0/441021261 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae38102c60 0x7fae38113980 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:49:55.634 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.999+0000 7fae3c977700 1 -- 172.21.15.6:0/441021261 >> 172.21.15.6:0/441021261 conn(0x7fae380fc580 msgr2=0x7fae38105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:49:55.634 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.999+0000 7fae3c977700 1 -- 172.21.15.6:0/441021261 shutdown_connections 2024-02-09T21:49:55.634 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:54.999+0000 7fae3c977700 1 -- 172.21.15.6:0/441021261 wait complete. 2024-02-09T21:49:55.634 INFO:teuthology.orchestra.run.smithi006.stdout:mgr not available, waiting (3/15)... 2024-02-09T21:49:56.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:55 smithi006 ceph-mon[29258]: from='mgr.14100 172.21.15.6:0/3743359400' entity='mgr.smithi006.glirxt' 2024-02-09T21:49:56.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:55 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/441021261' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-09T21:49:56.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:55 smithi006 ceph-mon[29258]: mgrmap e3: smithi006.glirxt(active, since 1.00681s) 2024-02-09T21:49:58.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:58 smithi006 ceph-mon[29258]: mgrmap e4: smithi006.glirxt(active, since 2s) 2024-02-09T21:49:58.825 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-02-09T21:49:58.825 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-02-09T21:49:58.825 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsid": "ee26d820-c794-11ee-95b8-87774f69a715", 2024-02-09T21:49:58.825 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "health": { 2024-02-09T21:49:58.826 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-09T21:49:58.826 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-09T21:49:58.826 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-09T21:49:58.826 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-02-09T21:49:58.826 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-09T21:49:58.826 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-09T21:49:58.826 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 0 2024-02-09T21:49:58.826 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-02-09T21:49:58.826 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-09T21:49:58.826 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "smithi006" 2024-02-09T21:49:58.826 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-02-09T21:49:58.826 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-02-09T21:49:58.826 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-09T21:49:58.826 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T21:49:58.826 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-02-09T21:49:58.827 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-09T21:49:58.827 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-02-09T21:49:58.827 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-09T21:49:58.827 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T21:49:58.827 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-09T21:49:58.827 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-09T21:49:58.835 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-09T21:49:58.835 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-09T21:49:58.836 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-09T21:49:58.836 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-09T21:49:58.836 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-02-09T21:49:58.836 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-09T21:49:58.836 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-09T21:49:58.836 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-09T21:49:58.836 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-09T21:49:58.836 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-09T21:49:58.836 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-09T21:49:58.836 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-09T21:49:58.836 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-09T21:49:58.836 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-09T21:49:58.836 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-02-09T21:49:58.836 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-09T21:49:58.837 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T21:49:58.837 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-09T21:49:58.837 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-09T21:49:58.837 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-02-09T21:49:58.837 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-09T21:49:58.837 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-09T21:49:58.837 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-09T21:49:58.837 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-09T21:49:58.837 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-09T21:49:58.837 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-09T21:49:58.837 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "restful" 2024-02-09T21:49:58.837 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-02-09T21:49:58.837 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-09T21:49:58.837 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-02-09T21:49:58.837 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-09T21:49:58.838 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T21:49:58.838 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modified": "2024-02-09T21:49:39.138636+0000", 2024-02-09T21:49:58.838 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-09T21:49:58.838 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-02-09T21:49:58.838 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-09T21:49:58.838 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-02-09T21:49:58.838 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.030+0000 7fb3f3f66700 1 Processor -- start 2024-02-09T21:49:58.838 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.031+0000 7fb3f3f66700 1 -- start start 2024-02-09T21:49:58.838 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.031+0000 7fb3f3f66700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3ec102770 0x7fb3ec102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:49:58.838 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.031+0000 7fb3f3f66700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec1030c0 con 0x7fb3ec102770 2024-02-09T21:49:58.838 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.031+0000 7fb3f1d02700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3ec102770 0x7fb3ec102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:49:58.838 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.031+0000 7fb3f1d02700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3ec102770 0x7fb3ec102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57184/0 (socket says 172.21.15.6:57184) 2024-02-09T21:49:58.838 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.031+0000 7fb3f1d02700 1 -- 172.21.15.6:0/869537423 learned_addr learned my addr 172.21.15.6:0/869537423 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:49:58.838 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.031+0000 7fb3f1d02700 1 -- 172.21.15.6:0/869537423 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ec103200 con 0x7fb3ec102770 2024-02-09T21:49:58.839 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.031+0000 7fb3f1d02700 1 --2- 172.21.15.6:0/869537423 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3ec102770 0x7fb3ec102b80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4009e50 tx=0x7fb3e4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9775278d3b1d3e4e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:49:58.839 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.032+0000 7fb3f0d00700 1 -- 172.21.15.6:0/869537423 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3e40092e0 con 0x7fb3ec102770 2024-02-09T21:49:58.839 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.032+0000 7fb3f0d00700 1 -- 172.21.15.6:0/869537423 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb3e4012430 con 0x7fb3ec102770 2024-02-09T21:49:58.839 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.032+0000 7fb3f0d00700 1 -- 172.21.15.6:0/869537423 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3e40075c0 con 0x7fb3ec102770 2024-02-09T21:49:58.839 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.032+0000 7fb3f3f66700 1 -- 172.21.15.6:0/869537423 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3ec102770 msgr2=0x7fb3ec102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:49:58.839 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.032+0000 7fb3f3f66700 1 --2- 172.21.15.6:0/869537423 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3ec102770 0x7fb3ec102b80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4009e50 tx=0x7fb3e4009840 comp rx=0 tx=0).stop 2024-02-09T21:49:58.839 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.032+0000 7fb3f3f66700 1 -- 172.21.15.6:0/869537423 shutdown_connections 2024-02-09T21:49:58.839 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.032+0000 7fb3f3f66700 1 --2- 172.21.15.6:0/869537423 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3ec102770 0x7fb3ec102b80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:49:58.839 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.032+0000 7fb3f3f66700 1 -- 172.21.15.6:0/869537423 >> 172.21.15.6:0/869537423 conn(0x7fb3ec0fded0 msgr2=0x7fb3ec100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:49:58.839 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.032+0000 7fb3f3f66700 1 -- 172.21.15.6:0/869537423 shutdown_connections 2024-02-09T21:49:58.839 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.032+0000 7fb3f3f66700 1 -- 172.21.15.6:0/869537423 wait complete. 2024-02-09T21:49:58.839 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.033+0000 7fb3f3f66700 1 Processor -- start 2024-02-09T21:49:58.839 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.033+0000 7fb3f3f66700 1 -- start start 2024-02-09T21:49:58.839 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.033+0000 7fb3f3f66700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3ec102770 0x7fb3ec1115b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:49:58.840 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.033+0000 7fb3f3f66700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec111af0 con 0x7fb3ec102770 2024-02-09T21:49:58.840 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.033+0000 7fb3f1d02700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3ec102770 0x7fb3ec1115b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:49:58.840 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.033+0000 7fb3f1d02700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3ec102770 0x7fb3ec1115b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57196/0 (socket says 172.21.15.6:57196) 2024-02-09T21:49:58.840 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.033+0000 7fb3f1d02700 1 -- 172.21.15.6:0/2384782044 learned_addr learned my addr 172.21.15.6:0/2384782044 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:49:58.840 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.033+0000 7fb3f1d02700 1 -- 172.21.15.6:0/2384782044 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e4009ae0 con 0x7fb3ec102770 2024-02-09T21:49:58.840 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.033+0000 7fb3f1d02700 1 --2- 172.21.15.6:0/2384782044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3ec102770 0x7fb3ec1115b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4009ab0 tx=0x7fb3e4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:49:58.840 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.034+0000 7fb3ea7fc700 1 -- 172.21.15.6:0/2384782044 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3e40092e0 con 0x7fb3ec102770 2024-02-09T21:49:58.840 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.034+0000 7fb3ea7fc700 1 -- 172.21.15.6:0/2384782044 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb3e4007d10 con 0x7fb3ec102770 2024-02-09T21:49:58.840 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.034+0000 7fb3f3f66700 1 -- 172.21.15.6:0/2384782044 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3ec111cf0 con 0x7fb3ec102770 2024-02-09T21:49:58.840 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.034+0000 7fb3ea7fc700 1 -- 172.21.15.6:0/2384782044 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3e401c990 con 0x7fb3ec102770 2024-02-09T21:49:58.840 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.034+0000 7fb3f3f66700 1 -- 172.21.15.6:0/2384782044 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3ec112990 con 0x7fb3ec102770 2024-02-09T21:49:58.840 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.034+0000 7fb3ea7fc700 1 -- 172.21.15.6:0/2384782044 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7fb3e4019070 con 0x7fb3ec102770 2024-02-09T21:49:58.840 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.034+0000 7fb3f3f66700 1 -- 172.21.15.6:0/2384782044 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3ec05f5c0 con 0x7fb3ec102770 2024-02-09T21:49:58.840 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.035+0000 7fb3ea7fc700 1 --2- 172.21.15.6:0/2384782044 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7fb3d4036740 0x7fb3d4038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:49:58.841 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.035+0000 7fb3ea7fc700 1 -- 172.21.15.6:0/2384782044 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fb3e4049f30 con 0x7fb3ec102770 2024-02-09T21:49:58.841 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.035+0000 7fb3f1501700 1 --2- 172.21.15.6:0/2384782044 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7fb3d4036740 0x7fb3d4038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:49:58.841 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.035+0000 7fb3f1501700 1 --2- 172.21.15.6:0/2384782044 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7fb3d4036740 0x7fb3d4038bf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb3e0009990 tx=0x7fb3e0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:49:58.841 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.040+0000 7fb3ea7fc700 1 -- 172.21.15.6:0/2384782044 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb3e40243b0 con 0x7fb3ec102770 2024-02-09T21:49:58.841 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.309+0000 7fb3f3f66700 1 -- 172.21.15.6:0/2384782044 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb3ec113500 con 0x7fb3ec102770 2024-02-09T21:49:58.841 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.309+0000 7fb3ea7fc700 1 -- 172.21.15.6:0/2384782044 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fb3e4017070 con 0x7fb3ec102770 2024-02-09T21:49:58.841 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.311+0000 7fb3f3f66700 1 -- 172.21.15.6:0/2384782044 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7fb3d4036740 msgr2=0x7fb3d4038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:49:58.841 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.311+0000 7fb3f3f66700 1 --2- 172.21.15.6:0/2384782044 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7fb3d4036740 0x7fb3d4038bf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb3e0009990 tx=0x7fb3e0006e40 comp rx=0 tx=0).stop 2024-02-09T21:49:58.841 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.311+0000 7fb3f3f66700 1 -- 172.21.15.6:0/2384782044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3ec102770 msgr2=0x7fb3ec1115b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:49:58.841 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.311+0000 7fb3f3f66700 1 --2- 172.21.15.6:0/2384782044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3ec102770 0x7fb3ec1115b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4009ab0 tx=0x7fb3e4009840 comp rx=0 tx=0).stop 2024-02-09T21:49:58.841 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.311+0000 7fb3f3f66700 1 -- 172.21.15.6:0/2384782044 shutdown_connections 2024-02-09T21:49:58.841 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.311+0000 7fb3f3f66700 1 --2- 172.21.15.6:0/2384782044 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7fb3d4036740 0x7fb3d4038bf0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:49:58.841 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.311+0000 7fb3f3f66700 1 --2- 172.21.15.6:0/2384782044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3ec102770 0x7fb3ec1115b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:49:58.841 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.311+0000 7fb3f3f66700 1 -- 172.21.15.6:0/2384782044 >> 172.21.15.6:0/2384782044 conn(0x7fb3ec0fded0 msgr2=0x7fb3ec0ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:49:58.841 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.311+0000 7fb3f3f66700 1 -- 172.21.15.6:0/2384782044 shutdown_connections 2024-02-09T21:49:58.842 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:58.311+0000 7fb3f3f66700 1 -- 172.21.15.6:0/2384782044 wait complete. 2024-02-09T21:49:58.842 INFO:teuthology.orchestra.run.smithi006.stdout:mgr is available 2024-02-09T21:49:59.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:49:59 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/2384782044' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-09T21:50:00.071 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-02-09T21:50:00.071 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout [global] 2024-02-09T21:50:00.071 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout fsid = ee26d820-c794-11ee-95b8-87774f69a715 2024-02-09T21:50:00.072 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-09T21:50:00.072 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-09T21:50:00.072 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-09T21:50:00.072 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-09T21:50:00.072 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-02-09T21:50:00.072 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-09T21:50:00.072 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-09T21:50:00.072 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-02-09T21:50:00.072 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout [osd] 2024-02-09T21:50:00.072 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-02-09T21:50:00.072 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-02-09T21:50:00.072 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-02-09T21:50:00.073 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-09T21:50:00.073 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-09T21:50:00.073 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.303+0000 7f444faae700 1 Processor -- start 2024-02-09T21:50:00.073 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.303+0000 7f444faae700 1 -- start start 2024-02-09T21:50:00.073 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.304+0000 7f444faae700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4448102770 0x7f4448102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:00.073 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.304+0000 7f444faae700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44481030c0 con 0x7f4448102770 2024-02-09T21:50:00.073 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.304+0000 7f444d84a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4448102770 0x7f4448102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:00.073 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.304+0000 7f444d84a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4448102770 0x7f4448102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57200/0 (socket says 172.21.15.6:57200) 2024-02-09T21:50:00.073 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.304+0000 7f444d84a700 1 -- 172.21.15.6:0/3476127565 learned_addr learned my addr 172.21.15.6:0/3476127565 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:00.073 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.304+0000 7f444d84a700 1 -- 172.21.15.6:0/3476127565 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4448103200 con 0x7f4448102770 2024-02-09T21:50:00.073 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.304+0000 7f444d84a700 1 --2- 172.21.15.6:0/3476127565 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4448102770 0x7f4448102b80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f44380098e0 tx=0x7f4438009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10508e69f029bb07 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:00.073 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.304+0000 7f444c848700 1 -- 172.21.15.6:0/3476127565 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f443800b860 con 0x7f4448102770 2024-02-09T21:50:00.074 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.304+0000 7f444c848700 1 -- 172.21.15.6:0/3476127565 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f443800be50 con 0x7f4448102770 2024-02-09T21:50:00.074 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.304+0000 7f444c848700 1 -- 172.21.15.6:0/3476127565 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4438010380 con 0x7f4448102770 2024-02-09T21:50:00.074 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.305+0000 7f444faae700 1 -- 172.21.15.6:0/3476127565 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4448102770 msgr2=0x7f4448102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:00.074 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.305+0000 7f444faae700 1 --2- 172.21.15.6:0/3476127565 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4448102770 0x7f4448102b80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f44380098e0 tx=0x7f4438009250 comp rx=0 tx=0).stop 2024-02-09T21:50:00.074 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.305+0000 7f444faae700 1 -- 172.21.15.6:0/3476127565 shutdown_connections 2024-02-09T21:50:00.074 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.305+0000 7f444faae700 1 --2- 172.21.15.6:0/3476127565 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4448102770 0x7f4448102b80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:00.074 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.305+0000 7f444faae700 1 -- 172.21.15.6:0/3476127565 >> 172.21.15.6:0/3476127565 conn(0x7f44480fded0 msgr2=0x7f4448100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:00.074 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.305+0000 7f444faae700 1 -- 172.21.15.6:0/3476127565 shutdown_connections 2024-02-09T21:50:00.074 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.305+0000 7f444faae700 1 -- 172.21.15.6:0/3476127565 wait complete. 2024-02-09T21:50:00.074 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.306+0000 7f444faae700 1 Processor -- start 2024-02-09T21:50:00.074 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.306+0000 7f444faae700 1 -- start start 2024-02-09T21:50:00.074 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.306+0000 7f444faae700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4448102770 0x7f4448111500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:00.074 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.306+0000 7f444faae700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4448111a40 con 0x7f4448102770 2024-02-09T21:50:00.075 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.306+0000 7f444d84a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4448102770 0x7f4448111500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:00.075 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.306+0000 7f444d84a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4448102770 0x7f4448111500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57216/0 (socket says 172.21.15.6:57216) 2024-02-09T21:50:00.075 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.306+0000 7f444d84a700 1 -- 172.21.15.6:0/1498983821 learned_addr learned my addr 172.21.15.6:0/1498983821 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:00.075 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.306+0000 7f444d84a700 1 -- 172.21.15.6:0/1498983821 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44380094f0 con 0x7f4448102770 2024-02-09T21:50:00.075 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.306+0000 7f444d84a700 1 --2- 172.21.15.6:0/1498983821 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4448102770 0x7f4448111500 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f443800b440 tx=0x7f4438009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:00.075 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.306+0000 7f443e7fc700 1 -- 172.21.15.6:0/1498983821 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4438010760 con 0x7f4448102770 2024-02-09T21:50:00.075 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.307+0000 7f444faae700 1 -- 172.21.15.6:0/1498983821 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4448111c40 con 0x7f4448102770 2024-02-09T21:50:00.075 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.307+0000 7f443e7fc700 1 -- 172.21.15.6:0/1498983821 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f4438010d50 con 0x7f4448102770 2024-02-09T21:50:00.075 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.307+0000 7f443e7fc700 1 -- 172.21.15.6:0/1498983821 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4438024d80 con 0x7f4448102770 2024-02-09T21:50:00.075 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.307+0000 7f444faae700 1 -- 172.21.15.6:0/1498983821 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44481128e0 con 0x7f4448102770 2024-02-09T21:50:00.075 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.307+0000 7f443e7fc700 1 -- 172.21.15.6:0/1498983821 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f443801a4b0 con 0x7f4448102770 2024-02-09T21:50:00.075 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.307+0000 7f443e7fc700 1 --2- 172.21.15.6:0/1498983821 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f44340366e0 0x7f4434038b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:00.075 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.308+0000 7f443e7fc700 1 -- 172.21.15.6:0/1498983821 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f443804a770 con 0x7f4448102770 2024-02-09T21:50:00.076 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.308+0000 7f444faae700 1 -- 172.21.15.6:0/1498983821 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f444805f5c0 con 0x7f4448102770 2024-02-09T21:50:00.076 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.308+0000 7f444d049700 1 --2- 172.21.15.6:0/1498983821 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f44340366e0 0x7f4434038b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:00.076 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.308+0000 7f444d049700 1 --2- 172.21.15.6:0/1498983821 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f44340366e0 0x7f4434038b90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4444009990 tx=0x7f4444006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:00.076 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.312+0000 7f443e7fc700 1 -- 172.21.15.6:0/1498983821 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f443802d3f0 con 0x7f4448102770 2024-02-09T21:50:00.076 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.507+0000 7f444faae700 1 -- 172.21.15.6:0/1498983821 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f444804bed0 con 0x7f4448102770 2024-02-09T21:50:00.076 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.508+0000 7f443e7fc700 1 -- 172.21.15.6:0/1498983821 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f4438018350 con 0x7f4448102770 2024-02-09T21:50:00.076 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.510+0000 7f444faae700 1 -- 172.21.15.6:0/1498983821 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f44340366e0 msgr2=0x7f4434038b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:00.076 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.510+0000 7f444faae700 1 --2- 172.21.15.6:0/1498983821 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f44340366e0 0x7f4434038b90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4444009990 tx=0x7f4444006e40 comp rx=0 tx=0).stop 2024-02-09T21:50:00.076 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.510+0000 7f444faae700 1 -- 172.21.15.6:0/1498983821 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4448102770 msgr2=0x7f4448111500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:00.076 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.510+0000 7f444faae700 1 --2- 172.21.15.6:0/1498983821 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4448102770 0x7f4448111500 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f443800b440 tx=0x7f4438009250 comp rx=0 tx=0).stop 2024-02-09T21:50:00.076 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.510+0000 7f444faae700 1 -- 172.21.15.6:0/1498983821 shutdown_connections 2024-02-09T21:50:00.077 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.510+0000 7f444faae700 1 --2- 172.21.15.6:0/1498983821 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f44340366e0 0x7f4434038b90 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:00.077 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.510+0000 7f444faae700 1 --2- 172.21.15.6:0/1498983821 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4448102770 0x7f4448111500 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:00.077 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.510+0000 7f444faae700 1 -- 172.21.15.6:0/1498983821 >> 172.21.15.6:0/1498983821 conn(0x7f44480fded0 msgr2=0x7f44480fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:00.077 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.510+0000 7f444faae700 1 -- 172.21.15.6:0/1498983821 shutdown_connections 2024-02-09T21:50:00.077 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:49:59.510+0000 7f444faae700 1 -- 172.21.15.6:0/1498983821 wait complete. 2024-02-09T21:50:00.077 INFO:teuthology.orchestra.run.smithi006.stdout:Enabling cephadm module... 2024-02-09T21:50:00.389 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:00 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/1498983821' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-09T21:50:01.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:01 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/1359661210' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-09T21:50:01.920 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.493+0000 7f0696c0e700 1 Processor -- start 2024-02-09T21:50:01.926 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.493+0000 7f0696c0e700 1 -- start start 2024-02-09T21:50:01.926 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.493+0000 7f0696c0e700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0690076c20 0x7f0690105380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:01.926 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.493+0000 7f0696c0e700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06901058c0 con 0x7f0690076c20 2024-02-09T21:50:01.926 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.493+0000 7f06949aa700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0690076c20 0x7f0690105380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:01.927 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.493+0000 7f06949aa700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0690076c20 0x7f0690105380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57230/0 (socket says 172.21.15.6:57230) 2024-02-09T21:50:01.927 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.493+0000 7f06949aa700 1 -- 172.21.15.6:0/479206560 learned_addr learned my addr 172.21.15.6:0/479206560 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:01.927 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.494+0000 7f06949aa700 1 -- 172.21.15.6:0/479206560 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0690105a00 con 0x7f0690076c20 2024-02-09T21:50:01.927 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.494+0000 7f06949aa700 1 --2- 172.21.15.6:0/479206560 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0690076c20 0x7f0690105380 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f06840098e0 tx=0x7f0684009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dab93c0c834931f5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:01.927 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.494+0000 7f068f7fe700 1 -- 172.21.15.6:0/479206560 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f068400b860 con 0x7f0690076c20 2024-02-09T21:50:01.927 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.494+0000 7f068f7fe700 1 -- 172.21.15.6:0/479206560 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f068400be50 con 0x7f0690076c20 2024-02-09T21:50:01.927 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.494+0000 7f068f7fe700 1 -- 172.21.15.6:0/479206560 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0684010380 con 0x7f0690076c20 2024-02-09T21:50:01.927 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.494+0000 7f0696c0e700 1 -- 172.21.15.6:0/479206560 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0690076c20 msgr2=0x7f0690105380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:01.927 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.494+0000 7f0696c0e700 1 --2- 172.21.15.6:0/479206560 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0690076c20 0x7f0690105380 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f06840098e0 tx=0x7f0684009250 comp rx=0 tx=0).stop 2024-02-09T21:50:01.927 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.494+0000 7f0696c0e700 1 -- 172.21.15.6:0/479206560 shutdown_connections 2024-02-09T21:50:01.927 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.494+0000 7f0696c0e700 1 --2- 172.21.15.6:0/479206560 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0690076c20 0x7f0690105380 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:01.927 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.495+0000 7f0696c0e700 1 -- 172.21.15.6:0/479206560 >> 172.21.15.6:0/479206560 conn(0x7f06900fcdd0 msgr2=0x7f06900ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:01.927 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.495+0000 7f0696c0e700 1 -- 172.21.15.6:0/479206560 shutdown_connections 2024-02-09T21:50:01.928 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.495+0000 7f0696c0e700 1 -- 172.21.15.6:0/479206560 wait complete. 2024-02-09T21:50:01.928 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.495+0000 7f0696c0e700 1 Processor -- start 2024-02-09T21:50:01.928 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.495+0000 7f0696c0e700 1 -- start start 2024-02-09T21:50:01.928 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.496+0000 7f0696c0e700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0690076c20 0x7f0690111830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:01.928 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.496+0000 7f0696c0e700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0690111d70 con 0x7f0690076c20 2024-02-09T21:50:01.928 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.496+0000 7f06949aa700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0690076c20 0x7f0690111830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:01.928 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.496+0000 7f06949aa700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0690076c20 0x7f0690111830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57242/0 (socket says 172.21.15.6:57242) 2024-02-09T21:50:01.928 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.496+0000 7f06949aa700 1 -- 172.21.15.6:0/1359661210 learned_addr learned my addr 172.21.15.6:0/1359661210 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:01.929 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.496+0000 7f06949aa700 1 -- 172.21.15.6:0/1359661210 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06840094f0 con 0x7f0690076c20 2024-02-09T21:50:01.929 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.496+0000 7f06949aa700 1 --2- 172.21.15.6:0/1359661210 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0690076c20 0x7f0690111830 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f068400b440 tx=0x7f0684009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:01.929 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.496+0000 7f068d7fa700 1 -- 172.21.15.6:0/1359661210 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06840107c0 con 0x7f0690076c20 2024-02-09T21:50:01.929 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.496+0000 7f0696c0e700 1 -- 172.21.15.6:0/1359661210 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0690111f70 con 0x7f0690076c20 2024-02-09T21:50:01.929 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.496+0000 7f068d7fa700 1 -- 172.21.15.6:0/1359661210 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f0684010db0 con 0x7f0690076c20 2024-02-09T21:50:01.929 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.496+0000 7f068d7fa700 1 -- 172.21.15.6:0/1359661210 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f068401ad30 con 0x7f0690076c20 2024-02-09T21:50:01.929 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.496+0000 7f0696c0e700 1 -- 172.21.15.6:0/1359661210 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0690112c10 con 0x7f0690076c20 2024-02-09T21:50:01.929 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.497+0000 7f068d7fa700 1 -- 172.21.15.6:0/1359661210 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f068402e440 con 0x7f0690076c20 2024-02-09T21:50:01.929 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.497+0000 7f0696c0e700 1 -- 172.21.15.6:0/1359661210 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f069005f5c0 con 0x7f0690076c20 2024-02-09T21:50:01.929 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.497+0000 7f068d7fa700 1 --2- 172.21.15.6:0/1359661210 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f0680036740 0x7f0680038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:01.929 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.497+0000 7f068d7fa700 1 -- 172.21.15.6:0/1359661210 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f068404ac80 con 0x7f0690076c20 2024-02-09T21:50:01.930 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.498+0000 7f068ffff700 1 --2- 172.21.15.6:0/1359661210 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f0680036740 0x7f0680038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:01.930 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.498+0000 7f068ffff700 1 --2- 172.21.15.6:0/1359661210 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f0680036740 0x7f0680038bf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f067c009990 tx=0x7f067c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:01.930 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.502+0000 7f068d7fa700 1 -- 172.21.15.6:0/1359661210 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f068402c980 con 0x7f0690076c20 2024-02-09T21:50:01.930 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:00.747+0000 7f0696c0e700 1 -- 172.21.15.6:0/1359661210 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f069004bed0 con 0x7f0690076c20 2024-02-09T21:50:01.930 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:01.296+0000 7f068d7fa700 1 -- 172.21.15.6:0/1359661210 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f068402e920 con 0x7f0690076c20 2024-02-09T21:50:01.930 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:01.297+0000 7f068d7fa700 1 -- 172.21.15.6:0/1359661210 <== mon.0 v2:172.21.15.6:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f068402c390 con 0x7f0690076c20 2024-02-09T21:50:01.930 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:01.300+0000 7f0696c0e700 1 -- 172.21.15.6:0/1359661210 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f0680036740 msgr2=0x7f0680038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:01.930 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:01.300+0000 7f0696c0e700 1 --2- 172.21.15.6:0/1359661210 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f0680036740 0x7f0680038bf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f067c009990 tx=0x7f067c006e40 comp rx=0 tx=0).stop 2024-02-09T21:50:01.930 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:01.300+0000 7f0696c0e700 1 -- 172.21.15.6:0/1359661210 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0690076c20 msgr2=0x7f0690111830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:01.930 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:01.300+0000 7f0696c0e700 1 --2- 172.21.15.6:0/1359661210 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0690076c20 0x7f0690111830 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f068400b440 tx=0x7f0684009250 comp rx=0 tx=0).stop 2024-02-09T21:50:01.930 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:01.300+0000 7f0696c0e700 1 -- 172.21.15.6:0/1359661210 shutdown_connections 2024-02-09T21:50:01.930 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:01.300+0000 7f0696c0e700 1 --2- 172.21.15.6:0/1359661210 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f0680036740 0x7f0680038bf0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:01.930 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:01.300+0000 7f0696c0e700 1 --2- 172.21.15.6:0/1359661210 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0690076c20 0x7f0690111830 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:01.931 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:01.300+0000 7f0696c0e700 1 -- 172.21.15.6:0/1359661210 >> 172.21.15.6:0/1359661210 conn(0x7f06900fcdd0 msgr2=0x7f06900fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:01.931 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:01.300+0000 7f0696c0e700 1 -- 172.21.15.6:0/1359661210 shutdown_connections 2024-02-09T21:50:01.931 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:01.300+0000 7f0696c0e700 1 -- 172.21.15.6:0/1359661210 wait complete. 2024-02-09T21:50:02.508 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:02 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/1359661210' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-09T21:50:02.508 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:02 smithi006 ceph-mon[29258]: mgrmap e5: smithi006.glirxt(active, since 6s) 2024-02-09T21:50:03.170 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-02-09T21:50:03.170 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-02-09T21:50:03.170 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-09T21:50:03.170 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "active_name": "smithi006.glirxt", 2024-02-09T21:50:03.170 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-09T21:50:03.170 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-02-09T21:50:03.170 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.299+0000 7f13b8e7e700 1 Processor -- start 2024-02-09T21:50:03.171 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.299+0000 7f13b8e7e700 1 -- start start 2024-02-09T21:50:03.171 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.299+0000 7f13b8e7e700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13b4101670 0x7f13b4101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:03.171 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.299+0000 7f13b8e7e700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b4101fc0 con 0x7f13b4101670 2024-02-09T21:50:03.171 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.300+0000 7f13b259c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13b4101670 0x7f13b4101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:03.171 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.300+0000 7f13b259c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13b4101670 0x7f13b4101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57256/0 (socket says 172.21.15.6:57256) 2024-02-09T21:50:03.171 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.300+0000 7f13b259c700 1 -- 172.21.15.6:0/3406569276 learned_addr learned my addr 172.21.15.6:0/3406569276 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:03.171 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.300+0000 7f13b259c700 1 -- 172.21.15.6:0/3406569276 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b4102100 con 0x7f13b4101670 2024-02-09T21:50:03.172 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.300+0000 7f13b259c700 1 --2- 172.21.15.6:0/3406569276 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13b4101670 0x7f13b4101a80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f13a400bb80 tx=0x7f13a400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1799f11d98e224e3 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:03.172 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.300+0000 7f13b159a700 1 -- 172.21.15.6:0/3406569276 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13a400dd00 con 0x7f13b4101670 2024-02-09T21:50:03.172 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.300+0000 7f13b159a700 1 -- 172.21.15.6:0/3406569276 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f13a400de60 con 0x7f13b4101670 2024-02-09T21:50:03.172 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.300+0000 7f13b159a700 1 -- 172.21.15.6:0/3406569276 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13a40188a0 con 0x7f13b4101670 2024-02-09T21:50:03.172 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.301+0000 7f13b8e7e700 1 -- 172.21.15.6:0/3406569276 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13b4101670 msgr2=0x7f13b4101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:03.172 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.301+0000 7f13b8e7e700 1 --2- 172.21.15.6:0/3406569276 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13b4101670 0x7f13b4101a80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f13a400bb80 tx=0x7f13a400b4f0 comp rx=0 tx=0).stop 2024-02-09T21:50:03.173 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.301+0000 7f13b8e7e700 1 -- 172.21.15.6:0/3406569276 shutdown_connections 2024-02-09T21:50:03.173 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.301+0000 7f13b8e7e700 1 --2- 172.21.15.6:0/3406569276 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13b4101670 0x7f13b4101a80 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:03.173 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.301+0000 7f13b8e7e700 1 -- 172.21.15.6:0/3406569276 >> 172.21.15.6:0/3406569276 conn(0x7f13b40fcdd0 msgr2=0x7f13b40ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:03.173 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.301+0000 7f13b8e7e700 1 -- 172.21.15.6:0/3406569276 shutdown_connections 2024-02-09T21:50:03.173 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.301+0000 7f13b8e7e700 1 -- 172.21.15.6:0/3406569276 wait complete. 2024-02-09T21:50:03.173 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.301+0000 7f13b8e7e700 1 Processor -- start 2024-02-09T21:50:03.173 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.302+0000 7f13b8e7e700 1 -- start start 2024-02-09T21:50:03.174 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.302+0000 7f13b8e7e700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13b4101670 0x7f13b4113710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:03.174 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.302+0000 7f13b8e7e700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b4113c50 con 0x7f13b4101670 2024-02-09T21:50:03.174 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.302+0000 7f13b259c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13b4101670 0x7f13b4113710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:03.174 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.302+0000 7f13b259c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13b4101670 0x7f13b4113710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57264/0 (socket says 172.21.15.6:57264) 2024-02-09T21:50:03.174 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.302+0000 7f13b259c700 1 -- 172.21.15.6:0/2849363979 learned_addr learned my addr 172.21.15.6:0/2849363979 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:03.174 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.302+0000 7f13b259c700 1 -- 172.21.15.6:0/2849363979 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a400b790 con 0x7f13b4101670 2024-02-09T21:50:03.175 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.302+0000 7f13b259c700 1 --2- 172.21.15.6:0/2849363979 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13b4101670 0x7f13b4113710 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f13a4010fa0 tx=0x7f13a400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:03.175 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.302+0000 7f13a2ffd700 1 -- 172.21.15.6:0/2849363979 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13a400dc70 con 0x7f13b4101670 2024-02-09T21:50:03.175 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.303+0000 7f13b8e7e700 1 -- 172.21.15.6:0/2849363979 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13b4113e50 con 0x7f13b4101670 2024-02-09T21:50:03.175 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.303+0000 7f13a2ffd700 1 -- 172.21.15.6:0/2849363979 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f13a4010bd0 con 0x7f13b4101670 2024-02-09T21:50:03.175 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.303+0000 7f13a2ffd700 1 -- 172.21.15.6:0/2849363979 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13a4021550 con 0x7f13b4101670 2024-02-09T21:50:03.175 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.303+0000 7f13b8e7e700 1 -- 172.21.15.6:0/2849363979 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13b4114af0 con 0x7f13b4101670 2024-02-09T21:50:03.176 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.304+0000 7f13a2ffd700 1 -- 172.21.15.6:0/2849363979 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f13a40216b0 con 0x7f13b4101670 2024-02-09T21:50:03.176 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.304+0000 7f13b8e7e700 1 -- 172.21.15.6:0/2849363979 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13b405f5c0 con 0x7f13b4101670 2024-02-09T21:50:03.176 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.304+0000 7f13a2ffd700 1 --2- 172.21.15.6:0/2849363979 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f139c0367e0 0x7f139c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:03.176 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.304+0000 7f13a2ffd700 1 -- 172.21.15.6:0/2849363979 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f13a4014070 con 0x7f13b4101670 2024-02-09T21:50:03.176 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.304+0000 7f13b1d9b700 1 -- 172.21.15.6:0/2849363979 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f139c0367e0 msgr2=0x7f139c038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/2325806844 2024-02-09T21:50:03.176 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.304+0000 7f13b1d9b700 1 --2- 172.21.15.6:0/2849363979 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f139c0367e0 0x7f139c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-09T21:50:03.177 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.313+0000 7f13a2ffd700 1 -- 172.21.15.6:0/2849363979 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f13a400e920 con 0x7f13b4101670 2024-02-09T21:50:03.177 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.505+0000 7f13b1d9b700 1 -- 172.21.15.6:0/2849363979 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f139c0367e0 msgr2=0x7f139c038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/2325806844 2024-02-09T21:50:03.177 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.505+0000 7f13b1d9b700 1 --2- 172.21.15.6:0/2849363979 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f139c0367e0 0x7f139c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-09T21:50:03.177 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.559+0000 7f13b8e7e700 1 -- 172.21.15.6:0/2849363979 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f13b4115600 con 0x7f13b4101670 2024-02-09T21:50:03.177 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.560+0000 7f13a2ffd700 1 -- 172.21.15.6:0/2849363979 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f13a4026070 con 0x7f13b4101670 2024-02-09T21:50:03.177 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.563+0000 7f13b8e7e700 1 -- 172.21.15.6:0/2849363979 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f139c0367e0 msgr2=0x7f139c038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-09T21:50:03.177 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.563+0000 7f13b8e7e700 1 --2- 172.21.15.6:0/2849363979 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f139c0367e0 0x7f139c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:03.178 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.563+0000 7f13b8e7e700 1 -- 172.21.15.6:0/2849363979 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13b4101670 msgr2=0x7f13b4113710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:03.178 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.563+0000 7f13b8e7e700 1 --2- 172.21.15.6:0/2849363979 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13b4101670 0x7f13b4113710 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f13a4010fa0 tx=0x7f13a400b4f0 comp rx=0 tx=0).stop 2024-02-09T21:50:03.178 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.563+0000 7f13b8e7e700 1 -- 172.21.15.6:0/2849363979 shutdown_connections 2024-02-09T21:50:03.179 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.563+0000 7f13b8e7e700 1 --2- 172.21.15.6:0/2849363979 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f139c0367e0 0x7f139c038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:03.179 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.563+0000 7f13b8e7e700 1 --2- 172.21.15.6:0/2849363979 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13b4101670 0x7f13b4113710 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:03.179 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.563+0000 7f13b8e7e700 1 -- 172.21.15.6:0/2849363979 >> 172.21.15.6:0/2849363979 conn(0x7f13b40fcdd0 msgr2=0x7f13b40ff180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:03.179 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.563+0000 7f13b8e7e700 1 -- 172.21.15.6:0/2849363979 shutdown_connections 2024-02-09T21:50:03.179 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:02.563+0000 7f13b8e7e700 1 -- 172.21.15.6:0/2849363979 wait complete. 2024-02-09T21:50:03.179 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for the mgr to restart... 2024-02-09T21:50:03.180 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for mgr epoch 5... 2024-02-09T21:50:03.468 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:03 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/2849363979' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-09T21:50:08.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:08 smithi006 ceph-mon[29258]: Active manager daemon smithi006.glirxt restarted 2024-02-09T21:50:08.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:08 smithi006 ceph-mon[29258]: Activating manager daemon smithi006.glirxt 2024-02-09T21:50:08.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:08 smithi006 ceph-mon[29258]: osdmap e2: 0 total, 0 up, 0 in 2024-02-09T21:50:08.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:08 smithi006 ceph-mon[29258]: mgrmap e6: smithi006.glirxt(active, starting, since 0.00319319s) 2024-02-09T21:50:08.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:08 smithi006 ceph-mon[29258]: from='mgr.14120 172.21.15.6:0/3931215362' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata", "id": "smithi006"}]: dispatch 2024-02-09T21:50:08.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:08 smithi006 ceph-mon[29258]: from='mgr.14120 172.21.15.6:0/3931215362' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mgr metadata", "who": "smithi006.glirxt", "id": "smithi006.glirxt"}]: dispatch 2024-02-09T21:50:08.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:08 smithi006 ceph-mon[29258]: from='mgr.14120 172.21.15.6:0/3931215362' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-09T21:50:08.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:08 smithi006 ceph-mon[29258]: from='mgr.14120 172.21.15.6:0/3931215362' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-09T21:50:08.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:08 smithi006 ceph-mon[29258]: from='mgr.14120 172.21.15.6:0/3931215362' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-09T21:50:08.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:08 smithi006 ceph-mon[29258]: Manager daemon smithi006.glirxt is now available 2024-02-09T21:50:08.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:08 smithi006 ceph-mon[29258]: from='mgr.14120 172.21.15.6:0/3931215362' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:08.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:08 smithi006 ceph-mon[29258]: from='mgr.14120 172.21.15.6:0/3931215362' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:08.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:08 smithi006 ceph-mon[29258]: from='mgr.14120 172.21.15.6:0/3931215362' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:50:08.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:08 smithi006 ceph-mon[29258]: from='mgr.14120 172.21.15.6:0/3931215362' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:50:08.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:08 smithi006 ceph-mon[29258]: from='mgr.14120 172.21.15.6:0/3931215362' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:50:09.305 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:09 smithi006 ceph-mon[29258]: Found migration_current of "None". Setting to last migration. 2024-02-09T21:50:09.305 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:09 smithi006 ceph-mon[29258]: from='mgr.14120 172.21.15.6:0/3931215362' entity='mgr.smithi006.glirxt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi006.glirxt/mirror_snapshot_schedule"}]: dispatch 2024-02-09T21:50:09.305 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:09 smithi006 ceph-mon[29258]: from='mgr.14120 172.21.15.6:0/3931215362' entity='mgr.smithi006.glirxt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi006.glirxt/trash_purge_schedule"}]: dispatch 2024-02-09T21:50:09.305 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:09 smithi006 ceph-mon[29258]: mgrmap e7: smithi006.glirxt(active, since 1.00578s) 2024-02-09T21:50:09.624 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-02-09T21:50:09.624 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-02-09T21:50:09.624 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-09T21:50:09.624 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-02-09T21:50:09.624 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.590+0000 7f3391c1e700 1 Processor -- start 2024-02-09T21:50:09.624 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.590+0000 7f3391c1e700 1 -- start start 2024-02-09T21:50:09.624 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.590+0000 7f3391c1e700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f338c071480 0x7f338c105380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:09.624 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.590+0000 7f3391c1e700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338c1058c0 con 0x7f338c071480 2024-02-09T21:50:09.624 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.591+0000 7f338b7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f338c071480 0x7f338c105380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:09.624 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.591+0000 7f338b7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f338c071480 0x7f338c105380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57274/0 (socket says 172.21.15.6:57274) 2024-02-09T21:50:09.625 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.591+0000 7f338b7fe700 1 -- 172.21.15.6:0/3933257934 learned_addr learned my addr 172.21.15.6:0/3933257934 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:09.625 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.591+0000 7f338b7fe700 1 -- 172.21.15.6:0/3933257934 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f338c105a00 con 0x7f338c071480 2024-02-09T21:50:09.625 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.591+0000 7f338b7fe700 1 --2- 172.21.15.6:0/3933257934 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f338c071480 0x7f338c105380 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f33800098e0 tx=0x7f3380009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8021d49c59c1f54 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:09.625 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.591+0000 7f338a7fc700 1 -- 172.21.15.6:0/3933257934 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f338000b860 con 0x7f338c071480 2024-02-09T21:50:09.625 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.591+0000 7f338a7fc700 1 -- 172.21.15.6:0/3933257934 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f338000be50 con 0x7f338c071480 2024-02-09T21:50:09.625 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.591+0000 7f338a7fc700 1 -- 172.21.15.6:0/3933257934 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3380010380 con 0x7f338c071480 2024-02-09T21:50:09.625 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.592+0000 7f3391c1e700 1 -- 172.21.15.6:0/3933257934 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f338c071480 msgr2=0x7f338c105380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:09.625 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.592+0000 7f3391c1e700 1 --2- 172.21.15.6:0/3933257934 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f338c071480 0x7f338c105380 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f33800098e0 tx=0x7f3380009250 comp rx=0 tx=0).stop 2024-02-09T21:50:09.625 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.592+0000 7f3391c1e700 1 -- 172.21.15.6:0/3933257934 shutdown_connections 2024-02-09T21:50:09.625 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.592+0000 7f3391c1e700 1 --2- 172.21.15.6:0/3933257934 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f338c071480 0x7f338c105380 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:09.625 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.592+0000 7f3391c1e700 1 -- 172.21.15.6:0/3933257934 >> 172.21.15.6:0/3933257934 conn(0x7f338c0fcdd0 msgr2=0x7f338c0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:09.625 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.592+0000 7f3391c1e700 1 -- 172.21.15.6:0/3933257934 shutdown_connections 2024-02-09T21:50:09.625 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.592+0000 7f3391c1e700 1 -- 172.21.15.6:0/3933257934 wait complete. 2024-02-09T21:50:09.625 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.593+0000 7f3391c1e700 1 Processor -- start 2024-02-09T21:50:09.625 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.593+0000 7f3391c1e700 1 -- start start 2024-02-09T21:50:09.626 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.593+0000 7f3391c1e700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f338c071480 0x7f338c102e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:09.626 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.593+0000 7f3391c1e700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338c101280 con 0x7f338c071480 2024-02-09T21:50:09.626 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.593+0000 7f338b7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f338c071480 0x7f338c102e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:09.626 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.593+0000 7f338b7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f338c071480 0x7f338c102e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57286/0 (socket says 172.21.15.6:57286) 2024-02-09T21:50:09.626 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.593+0000 7f338b7fe700 1 -- 172.21.15.6:0/2640169591 learned_addr learned my addr 172.21.15.6:0/2640169591 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:09.626 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.593+0000 7f338b7fe700 1 -- 172.21.15.6:0/2640169591 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33800094f0 con 0x7f338c071480 2024-02-09T21:50:09.626 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.593+0000 7f338b7fe700 1 --2- 172.21.15.6:0/2640169591 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f338c071480 0x7f338c102e00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3380003050 tx=0x7f3380009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:09.626 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.594+0000 7f3373fff700 1 -- 172.21.15.6:0/2640169591 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3380010760 con 0x7f338c071480 2024-02-09T21:50:09.626 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.594+0000 7f3391c1e700 1 -- 172.21.15.6:0/2640169591 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f338c101480 con 0x7f338c071480 2024-02-09T21:50:09.626 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.594+0000 7f3373fff700 1 -- 172.21.15.6:0/2640169591 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f3380010d50 con 0x7f338c071480 2024-02-09T21:50:09.626 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.594+0000 7f3373fff700 1 -- 172.21.15.6:0/2640169591 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f338001ae20 con 0x7f338c071480 2024-02-09T21:50:09.626 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.594+0000 7f3391c1e700 1 -- 172.21.15.6:0/2640169591 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f338c101910 con 0x7f338c071480 2024-02-09T21:50:09.626 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.594+0000 7f3373fff700 1 -- 172.21.15.6:0/2640169591 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f338002e440 con 0x7f338c071480 2024-02-09T21:50:09.626 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.595+0000 7f3373fff700 1 --2- 172.21.15.6:0/2640169591 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f3374036770 0x7f3374038c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:09.626 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.595+0000 7f3373fff700 1 -- 172.21.15.6:0/2640169591 --> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3374039330 con 0x7f3374036770 2024-02-09T21:50:09.627 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.595+0000 7f3373fff700 1 -- 172.21.15.6:0/2640169591 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f338004a4f0 con 0x7f338c071480 2024-02-09T21:50:09.627 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.595+0000 7f338affd700 1 -- 172.21.15.6:0/2640169591 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f3374036770 msgr2=0x7f3374038c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/2325806844 2024-02-09T21:50:09.627 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.595+0000 7f338affd700 1 --2- 172.21.15.6:0/2640169591 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f3374036770 0x7f3374038c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-09T21:50:09.627 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.796+0000 7f338affd700 1 -- 172.21.15.6:0/2640169591 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f3374036770 msgr2=0x7f3374038c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/2325806844 2024-02-09T21:50:09.627 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:03.796+0000 7f338affd700 1 --2- 172.21.15.6:0/2640169591 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f3374036770 0x7f3374038c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-09T21:50:09.627 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:04.197+0000 7f338affd700 1 -- 172.21.15.6:0/2640169591 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f3374036770 msgr2=0x7f3374038c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/2325806844 2024-02-09T21:50:09.627 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:04.197+0000 7f338affd700 1 --2- 172.21.15.6:0/2640169591 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f3374036770 0x7f3374038c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-09T21:50:09.627 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:04.998+0000 7f338affd700 1 -- 172.21.15.6:0/2640169591 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f3374036770 msgr2=0x7f3374038c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/2325806844 2024-02-09T21:50:09.627 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:04.998+0000 7f338affd700 1 --2- 172.21.15.6:0/2640169591 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f3374036770 0x7f3374038c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-09T21:50:09.627 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:06.599+0000 7f338affd700 1 -- 172.21.15.6:0/2640169591 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f3374036770 msgr2=0x7f3374038c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/2325806844 2024-02-09T21:50:09.627 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:06.599+0000 7f338affd700 1 --2- 172.21.15.6:0/2640169591 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f3374036770 0x7f3374038c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-02-09T21:50:09.627 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:07.988+0000 7f3373fff700 1 -- 172.21.15.6:0/2640169591 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mgrmap(e 6) v1 ==== 43135+0+0 (secure 0 0 0) 0x7f338002e880 con 0x7f338c071480 2024-02-09T21:50:09.627 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:07.988+0000 7f3373fff700 1 -- 172.21.15.6:0/2640169591 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f3374036770 msgr2=0x7f3374038c20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-09T21:50:09.627 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:07.988+0000 7f3373fff700 1 --2- 172.21.15.6:0/2640169591 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f3374036770 0x7f3374038c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:09.627 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:08.990+0000 7f3373fff700 1 -- 172.21.15.6:0/2640169591 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7f3380008d10 con 0x7f338c071480 2024-02-09T21:50:09.628 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:08.990+0000 7f3373fff700 1 --2- 172.21.15.6:0/2640169591 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f3374039ff0 0x7f337403c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:09.628 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:08.991+0000 7f3373fff700 1 -- 172.21.15.6:0/2640169591 --> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3374039330 con 0x7f3374039ff0 2024-02-09T21:50:09.628 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:08.991+0000 7f338affd700 1 --2- 172.21.15.6:0/2640169591 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f3374039ff0 0x7f337403c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:09.628 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:08.991+0000 7f338affd700 1 --2- 172.21.15.6:0/2640169591 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f3374039ff0 0x7f337403c3d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f337c0039d0 tx=0x7f337c009350 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:09.628 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:08.992+0000 7f3373fff700 1 -- 172.21.15.6:0/2640169591 <== mgr.14120 v2:172.21.15.6:6800/1422219104 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f3374039330 con 0x7f3374039ff0 2024-02-09T21:50:09.628 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:08.998+0000 7f3391c1e700 1 -- 172.21.15.6:0/2640169591 --> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f338c05f5c0 con 0x7f3374039ff0 2024-02-09T21:50:09.628 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:08.998+0000 7f3373fff700 1 -- 172.21.15.6:0/2640169591 <== mgr.14120 v2:172.21.15.6:6800/1422219104 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f338c05f5c0 con 0x7f3374039ff0 2024-02-09T21:50:09.628 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:08.999+0000 7f3391c1e700 1 -- 172.21.15.6:0/2640169591 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f3374039ff0 msgr2=0x7f337403c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:09.628 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:08.999+0000 7f3391c1e700 1 --2- 172.21.15.6:0/2640169591 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f3374039ff0 0x7f337403c3d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f337c0039d0 tx=0x7f337c009350 comp rx=0 tx=0).stop 2024-02-09T21:50:09.628 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:08.999+0000 7f3391c1e700 1 -- 172.21.15.6:0/2640169591 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f338c071480 msgr2=0x7f338c102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:09.628 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:08.999+0000 7f3391c1e700 1 --2- 172.21.15.6:0/2640169591 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f338c071480 0x7f338c102e00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3380003050 tx=0x7f3380009250 comp rx=0 tx=0).stop 2024-02-09T21:50:09.628 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:08.999+0000 7f3391c1e700 1 -- 172.21.15.6:0/2640169591 shutdown_connections 2024-02-09T21:50:09.628 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:08.999+0000 7f3391c1e700 1 --2- 172.21.15.6:0/2640169591 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f3374039ff0 0x7f337403c3d0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:09.628 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:08.999+0000 7f3391c1e700 1 --2- 172.21.15.6:0/2640169591 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f338c071480 0x7f338c102e00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:09.628 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:08.999+0000 7f3391c1e700 1 --2- 172.21.15.6:0/2640169591 >> [v2:172.21.15.6:6800/2325806844,v1:172.21.15.6:6801/2325806844] conn(0x7f3374036770 0x7f3374038c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:09.629 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:08.999+0000 7f3391c1e700 1 -- 172.21.15.6:0/2640169591 >> 172.21.15.6:0/2640169591 conn(0x7f338c0fcdd0 msgr2=0x7f338c0fe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:09.629 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:08.999+0000 7f3391c1e700 1 -- 172.21.15.6:0/2640169591 shutdown_connections 2024-02-09T21:50:09.629 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:08.999+0000 7f3391c1e700 1 -- 172.21.15.6:0/2640169591 wait complete. 2024-02-09T21:50:09.629 INFO:teuthology.orchestra.run.smithi006.stdout:mgr epoch 5 is available 2024-02-09T21:50:09.629 INFO:teuthology.orchestra.run.smithi006.stdout:Setting orchestrator backend to cephadm... 2024-02-09T21:50:10.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:10 smithi006 ceph-mon[29258]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-09T21:50:10.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:10 smithi006 ceph-mon[29258]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-09T21:50:10.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:10 smithi006 ceph-mon[29258]: from='mgr.14120 172.21.15.6:0/3931215362' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:10.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:10 smithi006 ceph-mon[29258]: from='mgr.14120 172.21.15.6:0/3931215362' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:50:10.861 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.063+0000 7f33f98dd700 1 Processor -- start 2024-02-09T21:50:10.861 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.064+0000 7f33f98dd700 1 -- start start 2024-02-09T21:50:10.862 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.064+0000 7f33f98dd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33f4101670 0x7f33f4101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:10.862 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.064+0000 7f33f98dd700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f4101fc0 con 0x7f33f4101670 2024-02-09T21:50:10.862 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.064+0000 7f33f2ffd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33f4101670 0x7f33f4101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:10.862 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.064+0000 7f33f2ffd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33f4101670 0x7f33f4101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52418/0 (socket says 172.21.15.6:52418) 2024-02-09T21:50:10.862 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.064+0000 7f33f2ffd700 1 -- 172.21.15.6:0/2146595715 learned_addr learned my addr 172.21.15.6:0/2146595715 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:10.862 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.064+0000 7f33f2ffd700 1 -- 172.21.15.6:0/2146595715 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f4102100 con 0x7f33f4101670 2024-02-09T21:50:10.862 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.064+0000 7f33f2ffd700 1 --2- 172.21.15.6:0/2146595715 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33f4101670 0x7f33f4101a80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f33dc0098e0 tx=0x7f33dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c3c0661609e1b4c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:10.862 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.064+0000 7f33f1ffb700 1 -- 172.21.15.6:0/2146595715 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33dc00b860 con 0x7f33f4101670 2024-02-09T21:50:10.862 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.064+0000 7f33f1ffb700 1 -- 172.21.15.6:0/2146595715 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f33dc00be50 con 0x7f33f4101670 2024-02-09T21:50:10.863 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.064+0000 7f33f1ffb700 1 -- 172.21.15.6:0/2146595715 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33dc010380 con 0x7f33f4101670 2024-02-09T21:50:10.863 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.065+0000 7f33f98dd700 1 -- 172.21.15.6:0/2146595715 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33f4101670 msgr2=0x7f33f4101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:10.863 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.065+0000 7f33f98dd700 1 --2- 172.21.15.6:0/2146595715 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33f4101670 0x7f33f4101a80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f33dc0098e0 tx=0x7f33dc009250 comp rx=0 tx=0).stop 2024-02-09T21:50:10.863 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.065+0000 7f33f98dd700 1 -- 172.21.15.6:0/2146595715 shutdown_connections 2024-02-09T21:50:10.863 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.065+0000 7f33f98dd700 1 --2- 172.21.15.6:0/2146595715 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33f4101670 0x7f33f4101a80 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:10.863 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.065+0000 7f33f98dd700 1 -- 172.21.15.6:0/2146595715 >> 172.21.15.6:0/2146595715 conn(0x7f33f40fcdd0 msgr2=0x7f33f40ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:10.863 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.065+0000 7f33f98dd700 1 -- 172.21.15.6:0/2146595715 shutdown_connections 2024-02-09T21:50:10.863 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.065+0000 7f33f98dd700 1 -- 172.21.15.6:0/2146595715 wait complete. 2024-02-09T21:50:10.863 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.066+0000 7f33f98dd700 1 Processor -- start 2024-02-09T21:50:10.863 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.066+0000 7f33f98dd700 1 -- start start 2024-02-09T21:50:10.863 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.066+0000 7f33f98dd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33f4101670 0x7f33f4114560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:10.863 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.066+0000 7f33f98dd700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f4076bf0 con 0x7f33f4101670 2024-02-09T21:50:10.863 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.066+0000 7f33f2ffd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33f4101670 0x7f33f4114560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:10.863 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.066+0000 7f33f2ffd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33f4101670 0x7f33f4114560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52430/0 (socket says 172.21.15.6:52430) 2024-02-09T21:50:10.863 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.066+0000 7f33f2ffd700 1 -- 172.21.15.6:0/2278288020 learned_addr learned my addr 172.21.15.6:0/2278288020 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:10.863 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.066+0000 7f33f2ffd700 1 -- 172.21.15.6:0/2278288020 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33dc0094f0 con 0x7f33f4101670 2024-02-09T21:50:10.864 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.066+0000 7f33f2ffd700 1 --2- 172.21.15.6:0/2278288020 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33f4101670 0x7f33f4114560 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f33dc00b440 tx=0x7f33dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:10.864 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.067+0000 7f33eb7fe700 1 -- 172.21.15.6:0/2278288020 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33dc010760 con 0x7f33f4101670 2024-02-09T21:50:10.864 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.067+0000 7f33eb7fe700 1 -- 172.21.15.6:0/2278288020 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f33dc010d50 con 0x7f33f4101670 2024-02-09T21:50:10.864 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.067+0000 7f33f98dd700 1 -- 172.21.15.6:0/2278288020 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33f4076df0 con 0x7f33f4101670 2024-02-09T21:50:10.864 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.067+0000 7f33eb7fe700 1 -- 172.21.15.6:0/2278288020 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33dc024d80 con 0x7f33f4101670 2024-02-09T21:50:10.864 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.067+0000 7f33f98dd700 1 -- 172.21.15.6:0/2278288020 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33f4114c60 con 0x7f33f4101670 2024-02-09T21:50:10.864 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.067+0000 7f33eb7fe700 1 -- 172.21.15.6:0/2278288020 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7f33dc01a430 con 0x7f33f4101670 2024-02-09T21:50:10.864 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.068+0000 7f33f98dd700 1 -- 172.21.15.6:0/2278288020 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33f410a570 con 0x7f33f4101670 2024-02-09T21:50:10.864 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.068+0000 7f33eb7fe700 1 --2- 172.21.15.6:0/2278288020 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f33e00366b0 0x7f33e0038b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:10.864 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.068+0000 7f33eb7fe700 1 -- 172.21.15.6:0/2278288020 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f33dc04a530 con 0x7f33f4101670 2024-02-09T21:50:10.864 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.068+0000 7f33f27fc700 1 --2- 172.21.15.6:0/2278288020 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f33e00366b0 0x7f33e0038b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:10.864 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.068+0000 7f33f27fc700 1 --2- 172.21.15.6:0/2278288020 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f33e00366b0 0x7f33e0038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f33e4009990 tx=0x7f33e4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:10.864 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.072+0000 7f33eb7fe700 1 -- 172.21.15.6:0/2278288020 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f33dc01a6e0 con 0x7f33f4101670 2024-02-09T21:50:10.864 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.283+0000 7f33f98dd700 1 -- 172.21.15.6:0/2278288020 --> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f33f405e3d0 con 0x7f33e00366b0 2024-02-09T21:50:10.865 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.289+0000 7f33eb7fe700 1 -- 172.21.15.6:0/2278288020 <== mgr.14120 v2:172.21.15.6:6800/1422219104 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f33f405e3d0 con 0x7f33e00366b0 2024-02-09T21:50:10.865 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.291+0000 7f33f98dd700 1 -- 172.21.15.6:0/2278288020 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f33e00366b0 msgr2=0x7f33e0038b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:10.865 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.291+0000 7f33f98dd700 1 --2- 172.21.15.6:0/2278288020 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f33e00366b0 0x7f33e0038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f33e4009990 tx=0x7f33e4006e40 comp rx=0 tx=0).stop 2024-02-09T21:50:10.865 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.291+0000 7f33f98dd700 1 -- 172.21.15.6:0/2278288020 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33f4101670 msgr2=0x7f33f4114560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:10.865 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.291+0000 7f33f98dd700 1 --2- 172.21.15.6:0/2278288020 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33f4101670 0x7f33f4114560 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f33dc00b440 tx=0x7f33dc009250 comp rx=0 tx=0).stop 2024-02-09T21:50:10.865 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.291+0000 7f33f98dd700 1 -- 172.21.15.6:0/2278288020 shutdown_connections 2024-02-09T21:50:10.865 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.291+0000 7f33f98dd700 1 --2- 172.21.15.6:0/2278288020 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f33e00366b0 0x7f33e0038b60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:10.865 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.291+0000 7f33f98dd700 1 --2- 172.21.15.6:0/2278288020 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33f4101670 0x7f33f4114560 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:10.865 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.291+0000 7f33f98dd700 1 -- 172.21.15.6:0/2278288020 >> 172.21.15.6:0/2278288020 conn(0x7f33f40fcdd0 msgr2=0x7f33f40fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:10.865 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.291+0000 7f33f98dd700 1 -- 172.21.15.6:0/2278288020 shutdown_connections 2024-02-09T21:50:10.865 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:10.291+0000 7f33f98dd700 1 -- 172.21.15.6:0/2278288020 wait complete. 2024-02-09T21:50:11.464 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:11 smithi006 ceph-mon[29258]: [09/Feb/2024:21:50:09] ENGINE Bus STARTING 2024-02-09T21:50:11.464 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:11 smithi006 ceph-mon[29258]: [09/Feb/2024:21:50:09] ENGINE Serving on https://172.21.15.6:7150 2024-02-09T21:50:11.465 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:11 smithi006 ceph-mon[29258]: [09/Feb/2024:21:50:09] ENGINE Bus STARTED 2024-02-09T21:50:11.465 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:11 smithi006 ceph-mon[29258]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:50:11.465 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:11 smithi006 ceph-mon[29258]: from='mgr.14120 172.21.15.6:0/3931215362' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:11.465 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:11 smithi006 ceph-mon[29258]: from='mgr.14120 172.21.15.6:0/3931215362' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:50:11.465 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:11 smithi006 ceph-mon[29258]: mgrmap e8: smithi006.glirxt(active, since 2s) 2024-02-09T21:50:11.943 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout value unchanged 2024-02-09T21:50:11.943 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.259+0000 7fa4e34b3700 1 Processor -- start 2024-02-09T21:50:11.943 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.259+0000 7fa4e34b3700 1 -- start start 2024-02-09T21:50:11.944 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.259+0000 7fa4e34b3700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4dc102770 0x7fa4dc102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:11.944 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.259+0000 7fa4e34b3700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4dc1030c0 con 0x7fa4dc102770 2024-02-09T21:50:11.944 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.259+0000 7fa4e124f700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4dc102770 0x7fa4dc102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:11.944 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.259+0000 7fa4e124f700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4dc102770 0x7fa4dc102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52438/0 (socket says 172.21.15.6:52438) 2024-02-09T21:50:11.944 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.259+0000 7fa4e124f700 1 -- 172.21.15.6:0/2546834226 learned_addr learned my addr 172.21.15.6:0/2546834226 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:11.944 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.260+0000 7fa4e124f700 1 -- 172.21.15.6:0/2546834226 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4dc103200 con 0x7fa4dc102770 2024-02-09T21:50:11.944 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.260+0000 7fa4e124f700 1 --2- 172.21.15.6:0/2546834226 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4dc102770 0x7fa4dc102b80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa4d80098e0 tx=0x7fa4d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cd784b3df5bb2 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:11.944 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.260+0000 7fa4d3fff700 1 -- 172.21.15.6:0/2546834226 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4d800b860 con 0x7fa4dc102770 2024-02-09T21:50:11.944 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.260+0000 7fa4d3fff700 1 -- 172.21.15.6:0/2546834226 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fa4d800be50 con 0x7fa4dc102770 2024-02-09T21:50:11.944 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.260+0000 7fa4d3fff700 1 -- 172.21.15.6:0/2546834226 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4d8010380 con 0x7fa4dc102770 2024-02-09T21:50:11.944 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.260+0000 7fa4e34b3700 1 -- 172.21.15.6:0/2546834226 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4dc102770 msgr2=0x7fa4dc102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:11.944 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.260+0000 7fa4e34b3700 1 --2- 172.21.15.6:0/2546834226 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4dc102770 0x7fa4dc102b80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa4d80098e0 tx=0x7fa4d8009250 comp rx=0 tx=0).stop 2024-02-09T21:50:11.944 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.260+0000 7fa4e34b3700 1 -- 172.21.15.6:0/2546834226 shutdown_connections 2024-02-09T21:50:11.944 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.260+0000 7fa4e34b3700 1 --2- 172.21.15.6:0/2546834226 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4dc102770 0x7fa4dc102b80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:11.945 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.260+0000 7fa4e34b3700 1 -- 172.21.15.6:0/2546834226 >> 172.21.15.6:0/2546834226 conn(0x7fa4dc0fded0 msgr2=0x7fa4dc100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:11.945 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.261+0000 7fa4e34b3700 1 -- 172.21.15.6:0/2546834226 shutdown_connections 2024-02-09T21:50:11.945 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.261+0000 7fa4e34b3700 1 -- 172.21.15.6:0/2546834226 wait complete. 2024-02-09T21:50:11.945 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.261+0000 7fa4e34b3700 1 Processor -- start 2024-02-09T21:50:11.945 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.261+0000 7fa4e34b3700 1 -- start start 2024-02-09T21:50:11.945 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.261+0000 7fa4e34b3700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4dc102770 0x7fa4dc078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:11.945 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.261+0000 7fa4e34b3700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4dc077190 con 0x7fa4dc102770 2024-02-09T21:50:11.945 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.261+0000 7fa4e124f700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4dc102770 0x7fa4dc078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:11.945 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.261+0000 7fa4e124f700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4dc102770 0x7fa4dc078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52444/0 (socket says 172.21.15.6:52444) 2024-02-09T21:50:11.945 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.261+0000 7fa4e124f700 1 -- 172.21.15.6:0/2916314685 learned_addr learned my addr 172.21.15.6:0/2916314685 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:11.945 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.262+0000 7fa4e124f700 1 -- 172.21.15.6:0/2916314685 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d80094f0 con 0x7fa4dc102770 2024-02-09T21:50:11.945 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.262+0000 7fa4e124f700 1 --2- 172.21.15.6:0/2916314685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4dc102770 0x7fa4dc078d10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa4d800b440 tx=0x7fa4d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:11.945 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.262+0000 7fa4d1ffb700 1 -- 172.21.15.6:0/2916314685 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4d8010760 con 0x7fa4dc102770 2024-02-09T21:50:11.945 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.262+0000 7fa4d1ffb700 1 -- 172.21.15.6:0/2916314685 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fa4d8010d50 con 0x7fa4dc102770 2024-02-09T21:50:11.945 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.262+0000 7fa4e34b3700 1 -- 172.21.15.6:0/2916314685 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4dc077390 con 0x7fa4dc102770 2024-02-09T21:50:11.946 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.262+0000 7fa4d1ffb700 1 -- 172.21.15.6:0/2916314685 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4d801ae20 con 0x7fa4dc102770 2024-02-09T21:50:11.946 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.262+0000 7fa4e34b3700 1 -- 172.21.15.6:0/2916314685 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4dc077820 con 0x7fa4dc102770 2024-02-09T21:50:11.946 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.263+0000 7fa4d1ffb700 1 -- 172.21.15.6:0/2916314685 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fa4d802e440 con 0x7fa4dc102770 2024-02-09T21:50:11.946 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.263+0000 7fa4d1ffb700 1 --2- 172.21.15.6:0/2916314685 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fa4c80367e0 0x7fa4c8038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:11.946 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.263+0000 7fa4e34b3700 1 -- 172.21.15.6:0/2916314685 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4dc04bed0 con 0x7fa4dc102770 2024-02-09T21:50:11.946 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.263+0000 7fa4d1ffb700 1 -- 172.21.15.6:0/2916314685 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa4d804a700 con 0x7fa4dc102770 2024-02-09T21:50:11.946 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.263+0000 7fa4e0a4e700 1 --2- 172.21.15.6:0/2916314685 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fa4c80367e0 0x7fa4c8038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:11.946 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.264+0000 7fa4e0a4e700 1 --2- 172.21.15.6:0/2916314685 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fa4c80367e0 0x7fa4c8038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc009990 tx=0x7fa4cc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:11.946 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.268+0000 7fa4d1ffb700 1 -- 172.21.15.6:0/2916314685 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa4d8018710 con 0x7fa4dc102770 2024-02-09T21:50:11.946 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.460+0000 7fa4e34b3700 1 -- 172.21.15.6:0/2916314685 --> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4dc078060 con 0x7fa4c80367e0 2024-02-09T21:50:11.946 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.461+0000 7fa4d1ffb700 1 -- 172.21.15.6:0/2916314685 <== mgr.14120 v2:172.21.15.6:6800/1422219104 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fa4dc078060 con 0x7fa4c80367e0 2024-02-09T21:50:11.946 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.463+0000 7fa4e34b3700 1 -- 172.21.15.6:0/2916314685 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fa4c80367e0 msgr2=0x7fa4c8038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:11.946 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.463+0000 7fa4e34b3700 1 --2- 172.21.15.6:0/2916314685 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fa4c80367e0 0x7fa4c8038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc009990 tx=0x7fa4cc006e40 comp rx=0 tx=0).stop 2024-02-09T21:50:11.946 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.463+0000 7fa4e34b3700 1 -- 172.21.15.6:0/2916314685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4dc102770 msgr2=0x7fa4dc078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:11.946 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.463+0000 7fa4e34b3700 1 --2- 172.21.15.6:0/2916314685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4dc102770 0x7fa4dc078d10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa4d800b440 tx=0x7fa4d8009250 comp rx=0 tx=0).stop 2024-02-09T21:50:11.946 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.463+0000 7fa4e34b3700 1 -- 172.21.15.6:0/2916314685 shutdown_connections 2024-02-09T21:50:11.947 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.463+0000 7fa4e34b3700 1 --2- 172.21.15.6:0/2916314685 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fa4c80367e0 0x7fa4c8038c90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:11.947 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.463+0000 7fa4e34b3700 1 --2- 172.21.15.6:0/2916314685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4dc102770 0x7fa4dc078d10 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:11.947 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.463+0000 7fa4e34b3700 1 -- 172.21.15.6:0/2916314685 >> 172.21.15.6:0/2916314685 conn(0x7fa4dc0fded0 msgr2=0x7fa4dc0ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:11.947 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.463+0000 7fa4e34b3700 1 -- 172.21.15.6:0/2916314685 shutdown_connections 2024-02-09T21:50:11.947 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:11.463+0000 7fa4e34b3700 1 -- 172.21.15.6:0/2916314685 wait complete. 2024-02-09T21:50:11.947 INFO:teuthology.orchestra.run.smithi006.stdout:Generating ssh key... 2024-02-09T21:50:12.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:12 smithi006 ceph-mon[29258]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:50:13.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:13 smithi006 ceph-mon[29258]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:50:13.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:13 smithi006 ceph-mon[29258]: Generating ssh key... 2024-02-09T21:50:14.250 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.369+0000 7f9d9b2da700 1 Processor -- start 2024-02-09T21:50:14.250 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.369+0000 7f9d9b2da700 1 -- start start 2024-02-09T21:50:14.251 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.369+0000 7f9d9b2da700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d94102770 0x7f9d94102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:14.251 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.369+0000 7f9d9b2da700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d941030c0 con 0x7f9d94102770 2024-02-09T21:50:14.251 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.369+0000 7f9d99076700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d94102770 0x7f9d94102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:14.251 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.369+0000 7f9d99076700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d94102770 0x7f9d94102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52460/0 (socket says 172.21.15.6:52460) 2024-02-09T21:50:14.251 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.369+0000 7f9d99076700 1 -- 172.21.15.6:0/38305297 learned_addr learned my addr 172.21.15.6:0/38305297 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:14.251 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.369+0000 7f9d99076700 1 -- 172.21.15.6:0/38305297 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d94103200 con 0x7f9d94102770 2024-02-09T21:50:14.251 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.370+0000 7f9d99076700 1 --2- 172.21.15.6:0/38305297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d94102770 0x7f9d94102b80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9d900098e0 tx=0x7f9d90009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d1c555c89dfb3cf server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:14.251 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.370+0000 7f9d8bfff700 1 -- 172.21.15.6:0/38305297 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d9000b860 con 0x7f9d94102770 2024-02-09T21:50:14.251 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.370+0000 7f9d8bfff700 1 -- 172.21.15.6:0/38305297 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f9d9000be50 con 0x7f9d94102770 2024-02-09T21:50:14.251 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.370+0000 7f9d8bfff700 1 -- 172.21.15.6:0/38305297 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d90010380 con 0x7f9d94102770 2024-02-09T21:50:14.251 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.370+0000 7f9d9b2da700 1 -- 172.21.15.6:0/38305297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d94102770 msgr2=0x7f9d94102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:14.251 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.370+0000 7f9d9b2da700 1 --2- 172.21.15.6:0/38305297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d94102770 0x7f9d94102b80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9d900098e0 tx=0x7f9d90009250 comp rx=0 tx=0).stop 2024-02-09T21:50:14.251 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.370+0000 7f9d9b2da700 1 -- 172.21.15.6:0/38305297 shutdown_connections 2024-02-09T21:50:14.251 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.370+0000 7f9d9b2da700 1 --2- 172.21.15.6:0/38305297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d94102770 0x7f9d94102b80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:14.251 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.370+0000 7f9d9b2da700 1 -- 172.21.15.6:0/38305297 >> 172.21.15.6:0/38305297 conn(0x7f9d940fded0 msgr2=0x7f9d94100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:14.252 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.371+0000 7f9d9b2da700 1 -- 172.21.15.6:0/38305297 shutdown_connections 2024-02-09T21:50:14.252 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.371+0000 7f9d9b2da700 1 -- 172.21.15.6:0/38305297 wait complete. 2024-02-09T21:50:14.252 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.371+0000 7f9d9b2da700 1 Processor -- start 2024-02-09T21:50:14.252 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.371+0000 7f9d9b2da700 1 -- start start 2024-02-09T21:50:14.252 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.371+0000 7f9d9b2da700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d94102770 0x7f9d94078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:14.252 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.371+0000 7f9d9b2da700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d94077190 con 0x7f9d94102770 2024-02-09T21:50:14.252 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.371+0000 7f9d99076700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d94102770 0x7f9d94078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:14.252 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.371+0000 7f9d99076700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d94102770 0x7f9d94078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52476/0 (socket says 172.21.15.6:52476) 2024-02-09T21:50:14.252 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.371+0000 7f9d99076700 1 -- 172.21.15.6:0/2866823170 learned_addr learned my addr 172.21.15.6:0/2866823170 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:14.252 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.372+0000 7f9d99076700 1 -- 172.21.15.6:0/2866823170 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d900094f0 con 0x7f9d94102770 2024-02-09T21:50:14.252 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.372+0000 7f9d99076700 1 --2- 172.21.15.6:0/2866823170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d94102770 0x7f9d94078d10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9d90000c00 tx=0x7f9d90009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:14.252 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.372+0000 7f9d89ffb700 1 -- 172.21.15.6:0/2866823170 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d900107c0 con 0x7f9d94102770 2024-02-09T21:50:14.252 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.372+0000 7f9d89ffb700 1 -- 172.21.15.6:0/2866823170 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f9d90010db0 con 0x7f9d94102770 2024-02-09T21:50:14.252 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.372+0000 7f9d9b2da700 1 -- 172.21.15.6:0/2866823170 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d94077390 con 0x7f9d94102770 2024-02-09T21:50:14.253 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.372+0000 7f9d89ffb700 1 -- 172.21.15.6:0/2866823170 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d9001ad00 con 0x7f9d94102770 2024-02-09T21:50:14.253 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.372+0000 7f9d9b2da700 1 -- 172.21.15.6:0/2866823170 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d94077820 con 0x7f9d94102770 2024-02-09T21:50:14.253 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.373+0000 7f9d89ffb700 1 -- 172.21.15.6:0/2866823170 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f9d900193c0 con 0x7f9d94102770 2024-02-09T21:50:14.253 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.373+0000 7f9d89ffb700 1 --2- 172.21.15.6:0/2866823170 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f9d80036790 0x7f9d80038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:14.253 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.373+0000 7f9d89ffb700 1 -- 172.21.15.6:0/2866823170 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f9d9004a6e0 con 0x7f9d94102770 2024-02-09T21:50:14.253 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.373+0000 7f9d98875700 1 --2- 172.21.15.6:0/2866823170 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f9d80036790 0x7f9d80038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:14.253 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.373+0000 7f9d9b2da700 1 -- 172.21.15.6:0/2866823170 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d9405f5c0 con 0x7f9d94102770 2024-02-09T21:50:14.253 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.374+0000 7f9d98875700 1 --2- 172.21.15.6:0/2866823170 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f9d80036790 0x7f9d80038c40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9d84009990 tx=0x7f9d84006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:14.253 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.378+0000 7f9d89ffb700 1 -- 172.21.15.6:0/2866823170 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9d900196c0 con 0x7f9d94102770 2024-02-09T21:50:14.253 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:12.572+0000 7f9d9b2da700 1 -- 172.21.15.6:0/2866823170 --> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d94077c90 con 0x7f9d80036790 2024-02-09T21:50:14.253 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:13.689+0000 7f9d89ffb700 1 -- 172.21.15.6:0/2866823170 <== mgr.14120 v2:172.21.15.6:6800/1422219104 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9d94077c90 con 0x7f9d80036790 2024-02-09T21:50:14.253 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:13.691+0000 7f9d9b2da700 1 -- 172.21.15.6:0/2866823170 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f9d80036790 msgr2=0x7f9d80038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:14.253 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:13.691+0000 7f9d9b2da700 1 --2- 172.21.15.6:0/2866823170 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f9d80036790 0x7f9d80038c40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9d84009990 tx=0x7f9d84006e40 comp rx=0 tx=0).stop 2024-02-09T21:50:14.253 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:13.691+0000 7f9d9b2da700 1 -- 172.21.15.6:0/2866823170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d94102770 msgr2=0x7f9d94078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:14.253 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:13.691+0000 7f9d9b2da700 1 --2- 172.21.15.6:0/2866823170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d94102770 0x7f9d94078d10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9d90000c00 tx=0x7f9d90009250 comp rx=0 tx=0).stop 2024-02-09T21:50:14.253 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:13.691+0000 7f9d9b2da700 1 -- 172.21.15.6:0/2866823170 shutdown_connections 2024-02-09T21:50:14.254 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:13.691+0000 7f9d9b2da700 1 --2- 172.21.15.6:0/2866823170 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f9d80036790 0x7f9d80038c40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:14.254 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:13.691+0000 7f9d9b2da700 1 --2- 172.21.15.6:0/2866823170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d94102770 0x7f9d94078d10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:14.254 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:13.691+0000 7f9d9b2da700 1 -- 172.21.15.6:0/2866823170 >> 172.21.15.6:0/2866823170 conn(0x7f9d940fded0 msgr2=0x7f9d940ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:14.254 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:13.691+0000 7f9d9b2da700 1 -- 172.21.15.6:0/2866823170 shutdown_connections 2024-02-09T21:50:14.254 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:13.691+0000 7f9d9b2da700 1 -- 172.21.15.6:0/2866823170 wait complete. 2024-02-09T21:50:14.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:14 smithi006 ceph-mon[29258]: from='mgr.14120 172.21.15.6:0/3931215362' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:14.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:14 smithi006 ceph-mon[29258]: from='mgr.14120 172.21.15.6:0/3931215362' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:15.441 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtYl/zJGGbFU4MjGUOF3yv3UUfjQxWFnzXGwhAabHxgwiZ4UbFBrCadqOsyTodexdSn+Ufxd8t60A6iqL+2so01tSSxAqr2xXw61tjkK1Q40+WnPI+kFHk/OvMtKHPy20Yl+jX9WUwd4CwcelzBrnmaZpR/pXvawnM2OWFkDRaoyJA3gK0m8YrHFdlvAFTwD8dsn79jGu9+xilARVrnNyBYGUwgeLQNFoGdlDanZHz8PX3q3XA/Tfl0fCRA/gTr8N5d4sYS7AMvEZY5cCBxXB/ZH8iYUltyPZg59/quEM3XPw8EcBzgl/ZmfPAncNDZWGftptFzPeb54i7qQOUc3Q9vITY7OnxGf36qeXJbpP5eGEOUlCfpbIREhiBJDKcjG9pBaxDrEPfVsEMbGC6FTrUVnI8TqyzyCCfmqQomJMpDr5TPtwq5GaYbtkz+NBLSVXTdqXDPRSqfvEX4U2EhDF6oTIf63fXyn+jUMdyk9O00LMzlUHE8Y6IVIx9wXzLO10= ceph-ee26d820-c794-11ee-95b8-87774f69a715 2024-02-09T21:50:15.441 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.615+0000 7fccb68f1700 1 Processor -- start 2024-02-09T21:50:15.442 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.615+0000 7fccb68f1700 1 -- start start 2024-02-09T21:50:15.442 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.615+0000 7fccb68f1700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccb0102780 0x7fccb0102b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:15.442 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.615+0000 7fccb68f1700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb01030d0 con 0x7fccb0102780 2024-02-09T21:50:15.442 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.615+0000 7fccaffff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccb0102780 0x7fccb0102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:15.442 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.615+0000 7fccaffff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccb0102780 0x7fccb0102b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52480/0 (socket says 172.21.15.6:52480) 2024-02-09T21:50:15.442 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.615+0000 7fccaffff700 1 -- 172.21.15.6:0/828871794 learned_addr learned my addr 172.21.15.6:0/828871794 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:15.442 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.615+0000 7fccaffff700 1 -- 172.21.15.6:0/828871794 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb0103210 con 0x7fccb0102780 2024-02-09T21:50:15.442 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.615+0000 7fccaffff700 1 --2- 172.21.15.6:0/828871794 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccb0102780 0x7fccb0102b90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcca00098e0 tx=0x7fcca0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5704a9f60a5362f7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:15.442 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.616+0000 7fccaeffd700 1 -- 172.21.15.6:0/828871794 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcca000b860 con 0x7fccb0102780 2024-02-09T21:50:15.442 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.616+0000 7fccaeffd700 1 -- 172.21.15.6:0/828871794 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fcca000be50 con 0x7fccb0102780 2024-02-09T21:50:15.442 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.616+0000 7fccaeffd700 1 -- 172.21.15.6:0/828871794 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcca0010380 con 0x7fccb0102780 2024-02-09T21:50:15.442 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.616+0000 7fccb68f1700 1 -- 172.21.15.6:0/828871794 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccb0102780 msgr2=0x7fccb0102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:15.442 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.616+0000 7fccb68f1700 1 --2- 172.21.15.6:0/828871794 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccb0102780 0x7fccb0102b90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcca00098e0 tx=0x7fcca0009250 comp rx=0 tx=0).stop 2024-02-09T21:50:15.442 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.616+0000 7fccb68f1700 1 -- 172.21.15.6:0/828871794 shutdown_connections 2024-02-09T21:50:15.442 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.616+0000 7fccb68f1700 1 --2- 172.21.15.6:0/828871794 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccb0102780 0x7fccb0102b90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:15.443 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.616+0000 7fccb68f1700 1 -- 172.21.15.6:0/828871794 >> 172.21.15.6:0/828871794 conn(0x7fccb00fdf00 msgr2=0x7fccb0100310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:15.443 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.616+0000 7fccb68f1700 1 -- 172.21.15.6:0/828871794 shutdown_connections 2024-02-09T21:50:15.443 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.616+0000 7fccb68f1700 1 -- 172.21.15.6:0/828871794 wait complete. 2024-02-09T21:50:15.443 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.617+0000 7fccb68f1700 1 Processor -- start 2024-02-09T21:50:15.443 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.617+0000 7fccb68f1700 1 -- start start 2024-02-09T21:50:15.443 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.617+0000 7fccb68f1700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccb0102780 0x7fccb0111560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:15.443 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.617+0000 7fccb68f1700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb0111aa0 con 0x7fccb0102780 2024-02-09T21:50:15.443 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.617+0000 7fccaffff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccb0102780 0x7fccb0111560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:15.443 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.617+0000 7fccaffff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccb0102780 0x7fccb0111560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52494/0 (socket says 172.21.15.6:52494) 2024-02-09T21:50:15.443 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.617+0000 7fccaffff700 1 -- 172.21.15.6:0/2283512747 learned_addr learned my addr 172.21.15.6:0/2283512747 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:15.443 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.618+0000 7fccaffff700 1 -- 172.21.15.6:0/2283512747 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca00094f0 con 0x7fccb0102780 2024-02-09T21:50:15.443 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.618+0000 7fccaffff700 1 --2- 172.21.15.6:0/2283512747 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccb0102780 0x7fccb0111560 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcca0000c00 tx=0x7fcca0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:15.443 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.618+0000 7fccacff9700 1 -- 172.21.15.6:0/2283512747 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcca00107c0 con 0x7fccb0102780 2024-02-09T21:50:15.443 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.618+0000 7fccb68f1700 1 -- 172.21.15.6:0/2283512747 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb0111ca0 con 0x7fccb0102780 2024-02-09T21:50:15.444 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.618+0000 7fccacff9700 1 -- 172.21.15.6:0/2283512747 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fcca0010db0 con 0x7fccb0102780 2024-02-09T21:50:15.444 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.618+0000 7fccacff9700 1 -- 172.21.15.6:0/2283512747 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcca001ad00 con 0x7fccb0102780 2024-02-09T21:50:15.444 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.618+0000 7fccb68f1700 1 -- 172.21.15.6:0/2283512747 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb0112940 con 0x7fccb0102780 2024-02-09T21:50:15.444 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.619+0000 7fccacff9700 1 -- 172.21.15.6:0/2283512747 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fcca00193c0 con 0x7fccb0102780 2024-02-09T21:50:15.444 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.619+0000 7fccacff9700 1 --2- 172.21.15.6:0/2283512747 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fcc98036790 0x7fcc98038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:15.444 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.619+0000 7fccacff9700 1 -- 172.21.15.6:0/2283512747 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fcca004a680 con 0x7fccb0102780 2024-02-09T21:50:15.444 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.619+0000 7fccb68f1700 1 -- 172.21.15.6:0/2283512747 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb005f5c0 con 0x7fccb0102780 2024-02-09T21:50:15.444 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.619+0000 7fccaf7fe700 1 --2- 172.21.15.6:0/2283512747 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fcc98036790 0x7fcc98038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:15.444 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.619+0000 7fccaf7fe700 1 --2- 172.21.15.6:0/2283512747 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fcc98036790 0x7fcc98038c40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fcca4009990 tx=0x7fcca4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:15.444 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.624+0000 7fccacff9700 1 -- 172.21.15.6:0/2283512747 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fcca00196c0 con 0x7fccb0102780 2024-02-09T21:50:15.444 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.818+0000 7fccb68f1700 1 -- 172.21.15.6:0/2283512747 --> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fccb00ff3f0 con 0x7fcc98036790 2024-02-09T21:50:15.444 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.818+0000 7fccacff9700 1 -- 172.21.15.6:0/2283512747 <== mgr.14120 v2:172.21.15.6:6800/1422219104 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fccb00ff3f0 con 0x7fcc98036790 2024-02-09T21:50:15.444 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.820+0000 7fccb68f1700 1 -- 172.21.15.6:0/2283512747 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fcc98036790 msgr2=0x7fcc98038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:15.444 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.820+0000 7fccb68f1700 1 --2- 172.21.15.6:0/2283512747 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fcc98036790 0x7fcc98038c40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fcca4009990 tx=0x7fcca4006e40 comp rx=0 tx=0).stop 2024-02-09T21:50:15.445 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.820+0000 7fccb68f1700 1 -- 172.21.15.6:0/2283512747 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccb0102780 msgr2=0x7fccb0111560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:15.445 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.820+0000 7fccb68f1700 1 --2- 172.21.15.6:0/2283512747 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccb0102780 0x7fccb0111560 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcca0000c00 tx=0x7fcca0009250 comp rx=0 tx=0).stop 2024-02-09T21:50:15.445 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.820+0000 7fccb68f1700 1 -- 172.21.15.6:0/2283512747 shutdown_connections 2024-02-09T21:50:15.445 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.820+0000 7fccb68f1700 1 --2- 172.21.15.6:0/2283512747 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fcc98036790 0x7fcc98038c40 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:15.445 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.820+0000 7fccb68f1700 1 --2- 172.21.15.6:0/2283512747 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fccb0102780 0x7fccb0111560 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:15.445 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.820+0000 7fccb68f1700 1 -- 172.21.15.6:0/2283512747 >> 172.21.15.6:0/2283512747 conn(0x7fccb00fdf00 msgr2=0x7fccb00fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:15.445 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.820+0000 7fccb68f1700 1 -- 172.21.15.6:0/2283512747 shutdown_connections 2024-02-09T21:50:15.445 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:14.820+0000 7fccb68f1700 1 -- 172.21.15.6:0/2283512747 wait complete. 2024-02-09T21:50:15.445 INFO:teuthology.orchestra.run.smithi006.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-09T21:50:15.445 INFO:teuthology.orchestra.run.smithi006.stdout:Adding key to root@localhost authorized_keys... 2024-02-09T21:50:15.445 INFO:teuthology.orchestra.run.smithi006.stdout:Adding host smithi006... 2024-02-09T21:50:16.027 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:15 smithi006 ceph-mon[29258]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:50:17.331 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:17 smithi006 ceph-mon[29258]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi006", "addr": "172.21.15.6", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:50:18.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:18 smithi006 ceph-mon[29258]: Deploying cephadm binary to smithi006 2024-02-09T21:50:18.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:18 smithi006 ceph-mon[29258]: from='mgr.14120 172.21.15.6:0/3931215362' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:18.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:18 smithi006 ceph-mon[29258]: from='mgr.14120 172.21.15.6:0/3931215362' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:50:18.682 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout Added host 'smithi006' with addr '172.21.15.6' 2024-02-09T21:50:18.682 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.819+0000 7f9aa5124700 1 Processor -- start 2024-02-09T21:50:18.682 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.819+0000 7f9aa5124700 1 -- start start 2024-02-09T21:50:18.682 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.819+0000 7f9aa5124700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aa0101670 0x7f9aa0101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:18.682 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.820+0000 7f9aa5124700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa0101fc0 con 0x7f9aa0101670 2024-02-09T21:50:18.682 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.820+0000 7f9a9ed9d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aa0101670 0x7f9aa0101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:18.682 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.820+0000 7f9a9ed9d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aa0101670 0x7f9aa0101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52508/0 (socket says 172.21.15.6:52508) 2024-02-09T21:50:18.682 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.820+0000 7f9a9ed9d700 1 -- 172.21.15.6:0/2126114461 learned_addr learned my addr 172.21.15.6:0/2126114461 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:18.682 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.820+0000 7f9a9ed9d700 1 -- 172.21.15.6:0/2126114461 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa0102100 con 0x7f9aa0101670 2024-02-09T21:50:18.683 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.820+0000 7f9a9ed9d700 1 --2- 172.21.15.6:0/2126114461 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aa0101670 0x7f9aa0101a80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9a940098e0 tx=0x7f9a94009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b9e468b59dc2195 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:18.683 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.820+0000 7f9a9dd9b700 1 -- 172.21.15.6:0/2126114461 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a9400b860 con 0x7f9aa0101670 2024-02-09T21:50:18.683 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.820+0000 7f9a9dd9b700 1 -- 172.21.15.6:0/2126114461 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f9a9400be50 con 0x7f9aa0101670 2024-02-09T21:50:18.683 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.820+0000 7f9a9dd9b700 1 -- 172.21.15.6:0/2126114461 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a94010380 con 0x7f9aa0101670 2024-02-09T21:50:18.683 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.821+0000 7f9aa5124700 1 -- 172.21.15.6:0/2126114461 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aa0101670 msgr2=0x7f9aa0101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:18.683 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.821+0000 7f9aa5124700 1 --2- 172.21.15.6:0/2126114461 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aa0101670 0x7f9aa0101a80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9a940098e0 tx=0x7f9a94009250 comp rx=0 tx=0).stop 2024-02-09T21:50:18.683 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.821+0000 7f9aa5124700 1 -- 172.21.15.6:0/2126114461 shutdown_connections 2024-02-09T21:50:18.683 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.821+0000 7f9aa5124700 1 --2- 172.21.15.6:0/2126114461 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aa0101670 0x7f9aa0101a80 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:18.683 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.821+0000 7f9aa5124700 1 -- 172.21.15.6:0/2126114461 >> 172.21.15.6:0/2126114461 conn(0x7f9aa00fcdd0 msgr2=0x7f9aa00ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:18.683 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.821+0000 7f9aa5124700 1 -- 172.21.15.6:0/2126114461 shutdown_connections 2024-02-09T21:50:18.683 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.821+0000 7f9aa5124700 1 -- 172.21.15.6:0/2126114461 wait complete. 2024-02-09T21:50:18.683 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.822+0000 7f9aa5124700 1 Processor -- start 2024-02-09T21:50:18.683 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.822+0000 7f9aa5124700 1 -- start start 2024-02-09T21:50:18.683 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.822+0000 7f9aa5124700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aa0101670 0x7f9aa0107a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:18.683 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.822+0000 7f9aa5124700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa0105f10 con 0x7f9aa0101670 2024-02-09T21:50:18.684 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.822+0000 7f9a9ed9d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aa0101670 0x7f9aa0107a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:18.684 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.822+0000 7f9a9ed9d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aa0101670 0x7f9aa0107a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52516/0 (socket says 172.21.15.6:52516) 2024-02-09T21:50:18.684 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.822+0000 7f9a9ed9d700 1 -- 172.21.15.6:0/3803990655 learned_addr learned my addr 172.21.15.6:0/3803990655 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:18.684 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.822+0000 7f9a9ed9d700 1 -- 172.21.15.6:0/3803990655 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a940094f0 con 0x7f9aa0101670 2024-02-09T21:50:18.684 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.822+0000 7f9a9ed9d700 1 --2- 172.21.15.6:0/3803990655 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aa0101670 0x7f9aa0107a90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9a9400b440 tx=0x7f9a94009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:18.684 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.822+0000 7f9a877fe700 1 -- 172.21.15.6:0/3803990655 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a940107c0 con 0x7f9aa0101670 2024-02-09T21:50:18.684 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.823+0000 7f9a877fe700 1 -- 172.21.15.6:0/3803990655 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f9a94010db0 con 0x7f9aa0101670 2024-02-09T21:50:18.684 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.823+0000 7f9aa5124700 1 -- 172.21.15.6:0/3803990655 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aa0106110 con 0x7f9aa0101670 2024-02-09T21:50:18.684 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.823+0000 7f9a877fe700 1 -- 172.21.15.6:0/3803990655 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a9401ada0 con 0x7f9aa0101670 2024-02-09T21:50:18.684 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.823+0000 7f9aa5124700 1 -- 172.21.15.6:0/3803990655 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aa01065a0 con 0x7f9aa0101670 2024-02-09T21:50:18.684 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.823+0000 7f9a877fe700 1 -- 172.21.15.6:0/3803990655 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f9a94010920 con 0x7f9aa0101670 2024-02-09T21:50:18.684 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.823+0000 7f9aa5124700 1 -- 172.21.15.6:0/3803990655 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aa010b770 con 0x7f9aa0101670 2024-02-09T21:50:18.684 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.824+0000 7f9a877fe700 1 --2- 172.21.15.6:0/3803990655 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f9a88036790 0x7f9a88038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:18.684 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.824+0000 7f9a877fe700 1 -- 172.21.15.6:0/3803990655 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f9a9404adb0 con 0x7f9aa0101670 2024-02-09T21:50:18.685 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.824+0000 7f9a9e59c700 1 --2- 172.21.15.6:0/3803990655 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f9a88036790 0x7f9a88038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:18.685 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.824+0000 7f9a9e59c700 1 --2- 172.21.15.6:0/3803990655 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f9a88036790 0x7f9a88038c40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9a90009990 tx=0x7f9a90006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:18.685 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:15.828+0000 7f9a877fe700 1 -- 172.21.15.6:0/3803990655 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9a9404f070 con 0x7f9aa0101670 2024-02-09T21:50:18.685 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:16.023+0000 7f9aa5124700 1 -- 172.21.15.6:0/3803990655 --> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi006", "addr": "172.21.15.6", "target": ["mon-mgr", ""]}) v1 -- 0x7f9aa0103650 con 0x7f9a88036790 2024-02-09T21:50:18.685 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:18.106+0000 7f9a877fe700 1 -- 172.21.15.6:0/3803990655 <== mgr.14120 v2:172.21.15.6:6800/1422219104 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f9aa0103650 con 0x7f9a88036790 2024-02-09T21:50:18.685 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:18.108+0000 7f9aa5124700 1 -- 172.21.15.6:0/3803990655 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f9a88036790 msgr2=0x7f9a88038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:18.685 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:18.108+0000 7f9aa5124700 1 --2- 172.21.15.6:0/3803990655 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f9a88036790 0x7f9a88038c40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9a90009990 tx=0x7f9a90006e40 comp rx=0 tx=0).stop 2024-02-09T21:50:18.685 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:18.109+0000 7f9aa5124700 1 -- 172.21.15.6:0/3803990655 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aa0101670 msgr2=0x7f9aa0107a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:18.685 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:18.109+0000 7f9aa5124700 1 --2- 172.21.15.6:0/3803990655 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aa0101670 0x7f9aa0107a90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9a9400b440 tx=0x7f9a94009250 comp rx=0 tx=0).stop 2024-02-09T21:50:18.685 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:18.109+0000 7f9aa5124700 1 -- 172.21.15.6:0/3803990655 shutdown_connections 2024-02-09T21:50:18.685 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:18.109+0000 7f9aa5124700 1 --2- 172.21.15.6:0/3803990655 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f9a88036790 0x7f9a88038c40 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:18.685 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:18.109+0000 7f9aa5124700 1 --2- 172.21.15.6:0/3803990655 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aa0101670 0x7f9aa0107a90 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:18.685 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:18.109+0000 7f9aa5124700 1 -- 172.21.15.6:0/3803990655 >> 172.21.15.6:0/3803990655 conn(0x7f9aa00fcdd0 msgr2=0x7f9aa00fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:18.685 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:18.109+0000 7f9aa5124700 1 -- 172.21.15.6:0/3803990655 shutdown_connections 2024-02-09T21:50:18.685 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:18.109+0000 7f9aa5124700 1 -- 172.21.15.6:0/3803990655 wait complete. 2024-02-09T21:50:18.686 INFO:teuthology.orchestra.run.smithi006.stdout:Deploying mon service with default placement... 2024-02-09T21:50:19.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:19 smithi006 ceph-mon[29258]: Added host smithi006 2024-02-09T21:50:20.192 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-02-09T21:50:20.192 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.237+0000 7f47a3abe700 1 Processor -- start 2024-02-09T21:50:20.193 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.237+0000 7f47a3abe700 1 -- start start 2024-02-09T21:50:20.193 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.237+0000 7f47a3abe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f479c102c40 0x7f479c103050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:20.193 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.237+0000 7f47a3abe700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479c103590 con 0x7f479c102c40 2024-02-09T21:50:20.193 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.237+0000 7f47a185a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f479c102c40 0x7f479c103050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:20.193 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.237+0000 7f47a185a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f479c102c40 0x7f479c103050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40992/0 (socket says 172.21.15.6:40992) 2024-02-09T21:50:20.193 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.237+0000 7f47a185a700 1 -- 172.21.15.6:0/140016861 learned_addr learned my addr 172.21.15.6:0/140016861 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:20.193 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.238+0000 7f47a185a700 1 -- 172.21.15.6:0/140016861 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f479c1036d0 con 0x7f479c102c40 2024-02-09T21:50:20.193 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.238+0000 7f47a185a700 1 --2- 172.21.15.6:0/140016861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f479c102c40 0x7f479c103050 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f47980098e0 tx=0x7f4798009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf8df495a334991a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:20.193 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.238+0000 7f47a0858700 1 -- 172.21.15.6:0/140016861 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f479800b860 con 0x7f479c102c40 2024-02-09T21:50:20.193 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.238+0000 7f47a0858700 1 -- 172.21.15.6:0/140016861 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f479800be50 con 0x7f479c102c40 2024-02-09T21:50:20.193 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.238+0000 7f47a0858700 1 -- 172.21.15.6:0/140016861 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4798010380 con 0x7f479c102c40 2024-02-09T21:50:20.193 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.239+0000 7f47a3abe700 1 -- 172.21.15.6:0/140016861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f479c102c40 msgr2=0x7f479c103050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:20.193 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.239+0000 7f47a3abe700 1 --2- 172.21.15.6:0/140016861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f479c102c40 0x7f479c103050 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f47980098e0 tx=0x7f4798009250 comp rx=0 tx=0).stop 2024-02-09T21:50:20.193 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.239+0000 7f47a3abe700 1 -- 172.21.15.6:0/140016861 shutdown_connections 2024-02-09T21:50:20.194 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.239+0000 7f47a3abe700 1 --2- 172.21.15.6:0/140016861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f479c102c40 0x7f479c103050 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:20.194 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.239+0000 7f47a3abe700 1 -- 172.21.15.6:0/140016861 >> 172.21.15.6:0/140016861 conn(0x7f479c0fc580 msgr2=0x7f479c0fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:20.194 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.239+0000 7f47a3abe700 1 -- 172.21.15.6:0/140016861 shutdown_connections 2024-02-09T21:50:20.194 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.239+0000 7f47a3abe700 1 -- 172.21.15.6:0/140016861 wait complete. 2024-02-09T21:50:20.194 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.239+0000 7f47a3abe700 1 Processor -- start 2024-02-09T21:50:20.194 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.239+0000 7f47a3abe700 1 -- start start 2024-02-09T21:50:20.194 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.239+0000 7f47a3abe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f479c102c40 0x7f479c10f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:20.194 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.239+0000 7f47a3abe700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479c10fb20 con 0x7f479c102c40 2024-02-09T21:50:20.194 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.240+0000 7f47a185a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f479c102c40 0x7f479c10f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:20.194 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.240+0000 7f47a185a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f479c102c40 0x7f479c10f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40998/0 (socket says 172.21.15.6:40998) 2024-02-09T21:50:20.194 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.240+0000 7f47a185a700 1 -- 172.21.15.6:0/3940654513 learned_addr learned my addr 172.21.15.6:0/3940654513 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:20.194 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.240+0000 7f47a185a700 1 -- 172.21.15.6:0/3940654513 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47980094f0 con 0x7f479c102c40 2024-02-09T21:50:20.194 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.240+0000 7f47a185a700 1 --2- 172.21.15.6:0/3940654513 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f479c102c40 0x7f479c10f5e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f479800b440 tx=0x7f4798009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:20.194 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.240+0000 7f47927fc700 1 -- 172.21.15.6:0/3940654513 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4798010760 con 0x7f479c102c40 2024-02-09T21:50:20.194 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.240+0000 7f47927fc700 1 -- 172.21.15.6:0/3940654513 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f4798010d50 con 0x7f479c102c40 2024-02-09T21:50:20.194 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.240+0000 7f47a3abe700 1 -- 172.21.15.6:0/3940654513 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f479c10fd20 con 0x7f479c102c40 2024-02-09T21:50:20.195 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.240+0000 7f47927fc700 1 -- 172.21.15.6:0/3940654513 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4798024d80 con 0x7f479c102c40 2024-02-09T21:50:20.195 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.240+0000 7f47a3abe700 1 -- 172.21.15.6:0/3940654513 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f479c1109c0 con 0x7f479c102c40 2024-02-09T21:50:20.195 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.241+0000 7f47927fc700 1 -- 172.21.15.6:0/3940654513 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f47980108c0 con 0x7f479c102c40 2024-02-09T21:50:20.195 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.241+0000 7f47a3abe700 1 -- 172.21.15.6:0/3940654513 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f479c05f5c0 con 0x7f479c102c40 2024-02-09T21:50:20.195 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.241+0000 7f47927fc700 1 --2- 172.21.15.6:0/3940654513 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f4788038a30 0x7f478803aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:20.195 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.241+0000 7f47927fc700 1 -- 172.21.15.6:0/3940654513 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f479804a680 con 0x7f479c102c40 2024-02-09T21:50:20.195 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.241+0000 7f47a1059700 1 --2- 172.21.15.6:0/3940654513 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f4788038a30 0x7f478803aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:20.195 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.242+0000 7f47a1059700 1 --2- 172.21.15.6:0/3940654513 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f4788038a30 0x7f478803aee0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f478c009990 tx=0x7f478c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:20.195 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.246+0000 7f47927fc700 1 -- 172.21.15.6:0/3940654513 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f479802d3f0 con 0x7f479c102c40 2024-02-09T21:50:20.195 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.510+0000 7f47a3abe700 1 -- 172.21.15.6:0/3940654513 --> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f479c102610 con 0x7f4788038a30 2024-02-09T21:50:20.195 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.514+0000 7f47927fc700 1 -- 172.21.15.6:0/3940654513 <== mgr.14120 v2:172.21.15.6:6800/1422219104 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f479c102610 con 0x7f4788038a30 2024-02-09T21:50:20.195 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.516+0000 7f47a3abe700 1 -- 172.21.15.6:0/3940654513 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f4788038a30 msgr2=0x7f478803aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:20.195 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.516+0000 7f47a3abe700 1 --2- 172.21.15.6:0/3940654513 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f4788038a30 0x7f478803aee0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f478c009990 tx=0x7f478c006e40 comp rx=0 tx=0).stop 2024-02-09T21:50:20.195 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.516+0000 7f47a3abe700 1 -- 172.21.15.6:0/3940654513 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f479c102c40 msgr2=0x7f479c10f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:20.195 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.516+0000 7f47a3abe700 1 --2- 172.21.15.6:0/3940654513 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f479c102c40 0x7f479c10f5e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f479800b440 tx=0x7f4798009250 comp rx=0 tx=0).stop 2024-02-09T21:50:20.196 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.516+0000 7f47a3abe700 1 -- 172.21.15.6:0/3940654513 shutdown_connections 2024-02-09T21:50:20.196 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.516+0000 7f47a3abe700 1 --2- 172.21.15.6:0/3940654513 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f4788038a30 0x7f478803aee0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:20.196 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.516+0000 7f47a3abe700 1 --2- 172.21.15.6:0/3940654513 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f479c102c40 0x7f479c10f5e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:20.196 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.516+0000 7f47a3abe700 1 -- 172.21.15.6:0/3940654513 >> 172.21.15.6:0/3940654513 conn(0x7f479c0fc580 msgr2=0x7f479c106470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:20.196 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.516+0000 7f47a3abe700 1 -- 172.21.15.6:0/3940654513 shutdown_connections 2024-02-09T21:50:20.196 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:19.516+0000 7f47a3abe700 1 -- 172.21.15.6:0/3940654513 wait complete. 2024-02-09T21:50:20.196 INFO:teuthology.orchestra.run.smithi006.stdout:Deploying mgr service with default placement... 2024-02-09T21:50:20.773 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:20 smithi006 ceph-mon[29258]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:50:20.773 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:20 smithi006 ceph-mon[29258]: Saving service mon spec with placement count:5 2024-02-09T21:50:20.773 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:20 smithi006 ceph-mon[29258]: from='mgr.14120 172.21.15.6:0/3931215362' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:21.660 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-02-09T21:50:21.660 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.568+0000 7ff2d6f6a700 1 Processor -- start 2024-02-09T21:50:21.660 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.568+0000 7ff2d6f6a700 1 -- start start 2024-02-09T21:50:21.660 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.568+0000 7ff2d6f6a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2d0102770 0x7ff2d0102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:21.660 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.568+0000 7ff2d6f6a700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d01030c0 con 0x7ff2d0102770 2024-02-09T21:50:21.661 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.568+0000 7ff2d4d06700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2d0102770 0x7ff2d0102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:21.661 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.568+0000 7ff2d4d06700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2d0102770 0x7ff2d0102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41014/0 (socket says 172.21.15.6:41014) 2024-02-09T21:50:21.661 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.568+0000 7ff2d4d06700 1 -- 172.21.15.6:0/4138158664 learned_addr learned my addr 172.21.15.6:0/4138158664 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:21.661 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.568+0000 7ff2d4d06700 1 -- 172.21.15.6:0/4138158664 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d0103200 con 0x7ff2d0102770 2024-02-09T21:50:21.661 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.569+0000 7ff2d4d06700 1 --2- 172.21.15.6:0/4138158664 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2d0102770 0x7ff2d0102b80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff2c40098e0 tx=0x7ff2c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=459fce19e6a9ede2 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:21.661 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.569+0000 7ff2cf7fe700 1 -- 172.21.15.6:0/4138158664 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2c400b860 con 0x7ff2d0102770 2024-02-09T21:50:21.661 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.569+0000 7ff2cf7fe700 1 -- 172.21.15.6:0/4138158664 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff2c400be50 con 0x7ff2d0102770 2024-02-09T21:50:21.661 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.569+0000 7ff2cf7fe700 1 -- 172.21.15.6:0/4138158664 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2c4010380 con 0x7ff2d0102770 2024-02-09T21:50:21.661 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.569+0000 7ff2d6f6a700 1 -- 172.21.15.6:0/4138158664 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2d0102770 msgr2=0x7ff2d0102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:21.661 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.569+0000 7ff2d6f6a700 1 --2- 172.21.15.6:0/4138158664 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2d0102770 0x7ff2d0102b80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff2c40098e0 tx=0x7ff2c4009250 comp rx=0 tx=0).stop 2024-02-09T21:50:21.661 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.569+0000 7ff2d6f6a700 1 -- 172.21.15.6:0/4138158664 shutdown_connections 2024-02-09T21:50:21.661 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.569+0000 7ff2d6f6a700 1 --2- 172.21.15.6:0/4138158664 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2d0102770 0x7ff2d0102b80 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:21.661 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.569+0000 7ff2d6f6a700 1 -- 172.21.15.6:0/4138158664 >> 172.21.15.6:0/4138158664 conn(0x7ff2d00fded0 msgr2=0x7ff2d0100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:21.662 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.569+0000 7ff2d6f6a700 1 -- 172.21.15.6:0/4138158664 shutdown_connections 2024-02-09T21:50:21.662 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.569+0000 7ff2d6f6a700 1 -- 172.21.15.6:0/4138158664 wait complete. 2024-02-09T21:50:21.662 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.570+0000 7ff2d6f6a700 1 Processor -- start 2024-02-09T21:50:21.662 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.570+0000 7ff2d6f6a700 1 -- start start 2024-02-09T21:50:21.662 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.570+0000 7ff2d6f6a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2d0102770 0x7ff2d0111560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:21.662 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.570+0000 7ff2d6f6a700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d0111aa0 con 0x7ff2d0102770 2024-02-09T21:50:21.662 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.570+0000 7ff2d4d06700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2d0102770 0x7ff2d0111560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:21.662 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.570+0000 7ff2d4d06700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2d0102770 0x7ff2d0111560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41016/0 (socket says 172.21.15.6:41016) 2024-02-09T21:50:21.662 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.570+0000 7ff2d4d06700 1 -- 172.21.15.6:0/58870563 learned_addr learned my addr 172.21.15.6:0/58870563 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:21.662 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.571+0000 7ff2d4d06700 1 -- 172.21.15.6:0/58870563 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c40094f0 con 0x7ff2d0102770 2024-02-09T21:50:21.662 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.571+0000 7ff2d4d06700 1 --2- 172.21.15.6:0/58870563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2d0102770 0x7ff2d0111560 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff2c400b440 tx=0x7ff2c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:21.662 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.571+0000 7ff2cd7fa700 1 -- 172.21.15.6:0/58870563 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2c4010760 con 0x7ff2d0102770 2024-02-09T21:50:21.662 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.571+0000 7ff2cd7fa700 1 -- 172.21.15.6:0/58870563 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff2c4010d50 con 0x7ff2d0102770 2024-02-09T21:50:21.662 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.571+0000 7ff2d6f6a700 1 -- 172.21.15.6:0/58870563 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2d0111ca0 con 0x7ff2d0102770 2024-02-09T21:50:21.662 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.571+0000 7ff2cd7fa700 1 -- 172.21.15.6:0/58870563 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2c401ae20 con 0x7ff2d0102770 2024-02-09T21:50:21.663 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.571+0000 7ff2d6f6a700 1 -- 172.21.15.6:0/58870563 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2d0112940 con 0x7ff2d0102770 2024-02-09T21:50:21.663 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.572+0000 7ff2cd7fa700 1 -- 172.21.15.6:0/58870563 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7ff2c40108c0 con 0x7ff2d0102770 2024-02-09T21:50:21.663 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.572+0000 7ff2cd7fa700 1 --2- 172.21.15.6:0/58870563 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7ff2c00367e0 0x7ff2c0038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:21.663 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.572+0000 7ff2d6f6a700 1 -- 172.21.15.6:0/58870563 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2d005f5c0 con 0x7ff2d0102770 2024-02-09T21:50:21.663 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.572+0000 7ff2cd7fa700 1 -- 172.21.15.6:0/58870563 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff2c404a550 con 0x7ff2d0102770 2024-02-09T21:50:21.663 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.572+0000 7ff2cffff700 1 --2- 172.21.15.6:0/58870563 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7ff2c00367e0 0x7ff2c0038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:21.663 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.572+0000 7ff2cffff700 1 --2- 172.21.15.6:0/58870563 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7ff2c00367e0 0x7ff2c0038c90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc009990 tx=0x7ff2bc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:21.663 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.577+0000 7ff2cd7fa700 1 -- 172.21.15.6:0/58870563 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff2c40187c0 con 0x7ff2d0102770 2024-02-09T21:50:21.663 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.787+0000 7ff2d6f6a700 1 -- 172.21.15.6:0/58870563 --> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7ff2d0100190 con 0x7ff2c00367e0 2024-02-09T21:50:21.663 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.792+0000 7ff2cd7fa700 1 -- 172.21.15.6:0/58870563 <== mgr.14120 v2:172.21.15.6:6800/1422219104 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7ff2d0100190 con 0x7ff2c00367e0 2024-02-09T21:50:21.663 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.794+0000 7ff2d6f6a700 1 -- 172.21.15.6:0/58870563 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7ff2c00367e0 msgr2=0x7ff2c0038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:21.663 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.794+0000 7ff2d6f6a700 1 --2- 172.21.15.6:0/58870563 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7ff2c00367e0 0x7ff2c0038c90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc009990 tx=0x7ff2bc006e40 comp rx=0 tx=0).stop 2024-02-09T21:50:21.663 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.794+0000 7ff2d6f6a700 1 -- 172.21.15.6:0/58870563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2d0102770 msgr2=0x7ff2d0111560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:21.663 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.794+0000 7ff2d6f6a700 1 --2- 172.21.15.6:0/58870563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2d0102770 0x7ff2d0111560 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff2c400b440 tx=0x7ff2c4009250 comp rx=0 tx=0).stop 2024-02-09T21:50:21.664 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.794+0000 7ff2d6f6a700 1 -- 172.21.15.6:0/58870563 shutdown_connections 2024-02-09T21:50:21.664 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.794+0000 7ff2d6f6a700 1 --2- 172.21.15.6:0/58870563 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7ff2c00367e0 0x7ff2c0038c90 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:21.664 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.794+0000 7ff2d6f6a700 1 --2- 172.21.15.6:0/58870563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2d0102770 0x7ff2d0111560 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:21.664 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.794+0000 7ff2d6f6a700 1 -- 172.21.15.6:0/58870563 >> 172.21.15.6:0/58870563 conn(0x7ff2d00fded0 msgr2=0x7ff2d00ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:21.664 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.794+0000 7ff2d6f6a700 1 -- 172.21.15.6:0/58870563 shutdown_connections 2024-02-09T21:50:21.664 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:20.794+0000 7ff2d6f6a700 1 -- 172.21.15.6:0/58870563 wait complete. 2024-02-09T21:50:21.664 INFO:teuthology.orchestra.run.smithi006.stdout:Deploying crash service with default placement... 2024-02-09T21:50:21.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:21 smithi006 ceph-mon[29258]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:50:21.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:21 smithi006 ceph-mon[29258]: Saving service mgr spec with placement count:2 2024-02-09T21:50:21.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:21 smithi006 ceph-mon[29258]: from='mgr.14120 172.21.15.6:0/3931215362' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:23.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:23 smithi006 ceph-mon[29258]: from='mgr.14120 172.21.15.6:0/3931215362' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:23.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:23 smithi006 ceph-mon[29258]: from='mgr.14120 172.21.15.6:0/3931215362' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:23.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:23 smithi006 ceph-mon[29258]: from='mgr.14120 172.21.15.6:0/3931215362' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:23.671 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-02-09T21:50:23.671 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.730+0000 7fee36be4700 1 Processor -- start 2024-02-09T21:50:23.671 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.730+0000 7fee36be4700 1 -- start start 2024-02-09T21:50:23.672 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.730+0000 7fee36be4700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee30076be0 0x7fee301077c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:23.672 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.730+0000 7fee36be4700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee301031b0 con 0x7fee30076be0 2024-02-09T21:50:23.672 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.730+0000 7fee34980700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee30076be0 0x7fee301077c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:23.672 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.730+0000 7fee34980700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee30076be0 0x7fee301077c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41020/0 (socket says 172.21.15.6:41020) 2024-02-09T21:50:23.672 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.730+0000 7fee34980700 1 -- 172.21.15.6:0/4030068817 learned_addr learned my addr 172.21.15.6:0/4030068817 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:23.672 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.731+0000 7fee34980700 1 -- 172.21.15.6:0/4030068817 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee30107d00 con 0x7fee30076be0 2024-02-09T21:50:23.672 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.731+0000 7fee34980700 1 --2- 172.21.15.6:0/4030068817 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee30076be0 0x7fee301077c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fee240098e0 tx=0x7fee24009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2210c23c6c766d9 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:23.672 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.731+0000 7fee2f7fe700 1 -- 172.21.15.6:0/4030068817 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee2400b860 con 0x7fee30076be0 2024-02-09T21:50:23.672 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.731+0000 7fee2f7fe700 1 -- 172.21.15.6:0/4030068817 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fee2400be90 con 0x7fee30076be0 2024-02-09T21:50:23.672 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.731+0000 7fee2f7fe700 1 -- 172.21.15.6:0/4030068817 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee240103c0 con 0x7fee30076be0 2024-02-09T21:50:23.672 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.732+0000 7fee36be4700 1 -- 172.21.15.6:0/4030068817 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee30076be0 msgr2=0x7fee301077c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:23.673 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.732+0000 7fee36be4700 1 --2- 172.21.15.6:0/4030068817 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee30076be0 0x7fee301077c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fee240098e0 tx=0x7fee24009250 comp rx=0 tx=0).stop 2024-02-09T21:50:23.673 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.732+0000 7fee36be4700 1 -- 172.21.15.6:0/4030068817 shutdown_connections 2024-02-09T21:50:23.673 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.732+0000 7fee36be4700 1 --2- 172.21.15.6:0/4030068817 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee30076be0 0x7fee301077c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:23.673 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.732+0000 7fee36be4700 1 -- 172.21.15.6:0/4030068817 >> 172.21.15.6:0/4030068817 conn(0x7fee300fcdd0 msgr2=0x7fee300ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:23.673 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.732+0000 7fee36be4700 1 -- 172.21.15.6:0/4030068817 shutdown_connections 2024-02-09T21:50:23.673 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.732+0000 7fee36be4700 1 -- 172.21.15.6:0/4030068817 wait complete. 2024-02-09T21:50:23.673 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.732+0000 7fee36be4700 1 Processor -- start 2024-02-09T21:50:23.673 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.732+0000 7fee36be4700 1 -- start start 2024-02-09T21:50:23.673 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.733+0000 7fee36be4700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee30076be0 0x7fee30113750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:23.673 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.733+0000 7fee36be4700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee30113c90 con 0x7fee30076be0 2024-02-09T21:50:23.673 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.733+0000 7fee34980700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee30076be0 0x7fee30113750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:23.673 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.733+0000 7fee34980700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee30076be0 0x7fee30113750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41032/0 (socket says 172.21.15.6:41032) 2024-02-09T21:50:23.673 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.733+0000 7fee34980700 1 -- 172.21.15.6:0/3587640803 learned_addr learned my addr 172.21.15.6:0/3587640803 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:23.673 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.733+0000 7fee34980700 1 -- 172.21.15.6:0/3587640803 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee240094f0 con 0x7fee30076be0 2024-02-09T21:50:23.673 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.733+0000 7fee34980700 1 --2- 172.21.15.6:0/3587640803 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee30076be0 0x7fee30113750 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fee24003080 tx=0x7fee24009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:23.674 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.733+0000 7fee2d7fa700 1 -- 172.21.15.6:0/3587640803 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee24011920 con 0x7fee30076be0 2024-02-09T21:50:23.674 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.733+0000 7fee2d7fa700 1 -- 172.21.15.6:0/3587640803 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fee24010d60 con 0x7fee30076be0 2024-02-09T21:50:23.674 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.733+0000 7fee36be4700 1 -- 172.21.15.6:0/3587640803 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee30113e90 con 0x7fee30076be0 2024-02-09T21:50:23.674 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.733+0000 7fee2d7fa700 1 -- 172.21.15.6:0/3587640803 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee240243f0 con 0x7fee30076be0 2024-02-09T21:50:23.674 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.734+0000 7fee36be4700 1 -- 172.21.15.6:0/3587640803 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee30114b30 con 0x7fee30076be0 2024-02-09T21:50:23.674 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.734+0000 7fee2d7fa700 1 -- 172.21.15.6:0/3587640803 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fee2402d3f0 con 0x7fee30076be0 2024-02-09T21:50:23.674 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.734+0000 7fee36be4700 1 -- 172.21.15.6:0/3587640803 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee3005f5c0 con 0x7fee30076be0 2024-02-09T21:50:23.674 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.735+0000 7fee2d7fa700 1 --2- 172.21.15.6:0/3587640803 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fee200367e0 0x7fee20038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:23.674 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.735+0000 7fee2d7fa700 1 -- 172.21.15.6:0/3587640803 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fee2404a540 con 0x7fee30076be0 2024-02-09T21:50:23.674 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.735+0000 7fee2ffff700 1 --2- 172.21.15.6:0/3587640803 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fee200367e0 0x7fee20038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:23.674 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.735+0000 7fee2ffff700 1 --2- 172.21.15.6:0/3587640803 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fee200367e0 0x7fee20038c90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fee1c009990 tx=0x7fee1c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:23.674 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.740+0000 7fee2d7fa700 1 -- 172.21.15.6:0/3587640803 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fee2402b8e0 con 0x7fee30076be0 2024-02-09T21:50:23.674 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.951+0000 7fee36be4700 1 -- 172.21.15.6:0/3587640803 --> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fee300ff0c0 con 0x7fee200367e0 2024-02-09T21:50:23.674 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.956+0000 7fee2d7fa700 1 -- 172.21.15.6:0/3587640803 <== mgr.14120 v2:172.21.15.6:6800/1422219104 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fee300ff0c0 con 0x7fee200367e0 2024-02-09T21:50:23.675 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.958+0000 7fee36be4700 1 -- 172.21.15.6:0/3587640803 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fee200367e0 msgr2=0x7fee20038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:23.675 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.958+0000 7fee36be4700 1 --2- 172.21.15.6:0/3587640803 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fee200367e0 0x7fee20038c90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fee1c009990 tx=0x7fee1c006e40 comp rx=0 tx=0).stop 2024-02-09T21:50:23.675 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.958+0000 7fee36be4700 1 -- 172.21.15.6:0/3587640803 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee30076be0 msgr2=0x7fee30113750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:23.675 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.958+0000 7fee36be4700 1 --2- 172.21.15.6:0/3587640803 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee30076be0 0x7fee30113750 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fee24003080 tx=0x7fee24009250 comp rx=0 tx=0).stop 2024-02-09T21:50:23.675 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.958+0000 7fee36be4700 1 -- 172.21.15.6:0/3587640803 shutdown_connections 2024-02-09T21:50:23.675 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.958+0000 7fee36be4700 1 --2- 172.21.15.6:0/3587640803 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fee200367e0 0x7fee20038c90 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:23.675 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.958+0000 7fee36be4700 1 --2- 172.21.15.6:0/3587640803 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee30076be0 0x7fee30113750 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:23.675 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.958+0000 7fee36be4700 1 -- 172.21.15.6:0/3587640803 >> 172.21.15.6:0/3587640803 conn(0x7fee300fcdd0 msgr2=0x7fee300fe4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:23.675 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.958+0000 7fee36be4700 1 -- 172.21.15.6:0/3587640803 shutdown_connections 2024-02-09T21:50:23.675 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:22.958+0000 7fee36be4700 1 -- 172.21.15.6:0/3587640803 wait complete. 2024-02-09T21:50:23.675 INFO:teuthology.orchestra.run.smithi006.stdout:Deploying prometheus service with default placement... 2024-02-09T21:50:24.630 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:24 smithi006 ceph-mon[29258]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:50:24.631 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:24 smithi006 ceph-mon[29258]: Saving service crash spec with placement * 2024-02-09T21:50:24.631 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:24 smithi006 ceph-mon[29258]: from='mgr.14120 172.21.15.6:0/3931215362' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:25.189 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-02-09T21:50:25.189 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.069+0000 7f468af5e700 1 Processor -- start 2024-02-09T21:50:25.189 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.069+0000 7f468af5e700 1 -- start start 2024-02-09T21:50:25.190 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.069+0000 7f468af5e700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4684102770 0x7f4684102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:25.190 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.069+0000 7f468af5e700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46841030c0 con 0x7f4684102770 2024-02-09T21:50:25.190 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.070+0000 7f4688cfa700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4684102770 0x7f4684102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:25.190 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.070+0000 7f4688cfa700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4684102770 0x7f4684102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41046/0 (socket says 172.21.15.6:41046) 2024-02-09T21:50:25.190 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.070+0000 7f4688cfa700 1 -- 172.21.15.6:0/142652981 learned_addr learned my addr 172.21.15.6:0/142652981 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:25.190 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.070+0000 7f4688cfa700 1 -- 172.21.15.6:0/142652981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4684103200 con 0x7f4684102770 2024-02-09T21:50:25.190 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.070+0000 7f4688cfa700 1 --2- 172.21.15.6:0/142652981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4684102770 0x7f4684102b80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f46780098e0 tx=0x7f4678009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0336a30be374c0a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:25.190 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.070+0000 7f46837fe700 1 -- 172.21.15.6:0/142652981 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f467800b860 con 0x7f4684102770 2024-02-09T21:50:25.190 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.070+0000 7f46837fe700 1 -- 172.21.15.6:0/142652981 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f467800be90 con 0x7f4684102770 2024-02-09T21:50:25.190 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.070+0000 7f46837fe700 1 -- 172.21.15.6:0/142652981 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46780103c0 con 0x7f4684102770 2024-02-09T21:50:25.190 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.071+0000 7f468af5e700 1 -- 172.21.15.6:0/142652981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4684102770 msgr2=0x7f4684102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:25.190 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.071+0000 7f468af5e700 1 --2- 172.21.15.6:0/142652981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4684102770 0x7f4684102b80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f46780098e0 tx=0x7f4678009250 comp rx=0 tx=0).stop 2024-02-09T21:50:25.190 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.071+0000 7f468af5e700 1 -- 172.21.15.6:0/142652981 shutdown_connections 2024-02-09T21:50:25.191 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.071+0000 7f468af5e700 1 --2- 172.21.15.6:0/142652981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4684102770 0x7f4684102b80 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:25.191 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.071+0000 7f468af5e700 1 -- 172.21.15.6:0/142652981 >> 172.21.15.6:0/142652981 conn(0x7f46840fded0 msgr2=0x7f4684100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:25.191 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.071+0000 7f468af5e700 1 -- 172.21.15.6:0/142652981 shutdown_connections 2024-02-09T21:50:25.191 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.071+0000 7f468af5e700 1 -- 172.21.15.6:0/142652981 wait complete. 2024-02-09T21:50:25.191 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.071+0000 7f468af5e700 1 Processor -- start 2024-02-09T21:50:25.191 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.071+0000 7f468af5e700 1 -- start start 2024-02-09T21:50:25.191 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.072+0000 7f468af5e700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4684102770 0x7f4684077c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:25.191 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.072+0000 7f468af5e700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4684076080 con 0x7f4684102770 2024-02-09T21:50:25.191 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.072+0000 7f4688cfa700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4684102770 0x7f4684077c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:25.191 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.072+0000 7f4688cfa700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4684102770 0x7f4684077c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41048/0 (socket says 172.21.15.6:41048) 2024-02-09T21:50:25.191 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.072+0000 7f4688cfa700 1 -- 172.21.15.6:0/2328672760 learned_addr learned my addr 172.21.15.6:0/2328672760 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:25.191 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.072+0000 7f4688cfa700 1 -- 172.21.15.6:0/2328672760 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46780094f0 con 0x7f4684102770 2024-02-09T21:50:25.191 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.072+0000 7f4688cfa700 1 --2- 172.21.15.6:0/2328672760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4684102770 0x7f4684077c00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4678011650 tx=0x7f4678009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:25.191 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.072+0000 7f46817fa700 1 -- 172.21.15.6:0/2328672760 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4678010890 con 0x7f4684102770 2024-02-09T21:50:25.191 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.072+0000 7f468af5e700 1 -- 172.21.15.6:0/2328672760 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4684076280 con 0x7f4684102770 2024-02-09T21:50:25.192 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.072+0000 7f46817fa700 1 -- 172.21.15.6:0/2328672760 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4678010ec0 con 0x7f4684102770 2024-02-09T21:50:25.192 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.072+0000 7f46817fa700 1 -- 172.21.15.6:0/2328672760 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f467801a430 con 0x7f4684102770 2024-02-09T21:50:25.192 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.072+0000 7f468af5e700 1 -- 172.21.15.6:0/2328672760 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4684076710 con 0x7f4684102770 2024-02-09T21:50:25.192 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.073+0000 7f46817fa700 1 -- 172.21.15.6:0/2328672760 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f467801a6b0 con 0x7f4684102770 2024-02-09T21:50:25.192 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.073+0000 7f468af5e700 1 -- 172.21.15.6:0/2328672760 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f468410b690 con 0x7f4684102770 2024-02-09T21:50:25.192 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.073+0000 7f46817fa700 1 --2- 172.21.15.6:0/2328672760 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f4674036790 0x7f4674038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:25.192 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.073+0000 7f46817fa700 1 -- 172.21.15.6:0/2328672760 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f467804a5f0 con 0x7f4684102770 2024-02-09T21:50:25.192 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.074+0000 7f4683fff700 1 --2- 172.21.15.6:0/2328672760 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f4674036790 0x7f4674038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:25.192 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.074+0000 7f4683fff700 1 --2- 172.21.15.6:0/2328672760 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f4674036790 0x7f4674038c40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4670009990 tx=0x7f4670006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:25.192 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.078+0000 7f46817fa700 1 -- 172.21.15.6:0/2328672760 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4678018710 con 0x7f4684102770 2024-02-09T21:50:25.192 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.280+0000 7f468af5e700 1 -- 172.21.15.6:0/2328672760 --> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f4684104720 con 0x7f4674036790 2024-02-09T21:50:25.192 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.284+0000 7f46817fa700 1 -- 172.21.15.6:0/2328672760 <== mgr.14120 v2:172.21.15.6:6800/1422219104 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f4684104720 con 0x7f4674036790 2024-02-09T21:50:25.192 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.286+0000 7f468af5e700 1 -- 172.21.15.6:0/2328672760 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f4674036790 msgr2=0x7f4674038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:25.193 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.286+0000 7f468af5e700 1 --2- 172.21.15.6:0/2328672760 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f4674036790 0x7f4674038c40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4670009990 tx=0x7f4670006e40 comp rx=0 tx=0).stop 2024-02-09T21:50:25.193 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.286+0000 7f468af5e700 1 -- 172.21.15.6:0/2328672760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4684102770 msgr2=0x7f4684077c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:25.193 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.286+0000 7f468af5e700 1 --2- 172.21.15.6:0/2328672760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4684102770 0x7f4684077c00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4678011650 tx=0x7f4678009250 comp rx=0 tx=0).stop 2024-02-09T21:50:25.193 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.286+0000 7f468af5e700 1 -- 172.21.15.6:0/2328672760 shutdown_connections 2024-02-09T21:50:25.193 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.286+0000 7f468af5e700 1 --2- 172.21.15.6:0/2328672760 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f4674036790 0x7f4674038c40 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:25.193 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.286+0000 7f468af5e700 1 --2- 172.21.15.6:0/2328672760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4684102770 0x7f4684077c00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:25.193 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.286+0000 7f468af5e700 1 -- 172.21.15.6:0/2328672760 >> 172.21.15.6:0/2328672760 conn(0x7f46840fded0 msgr2=0x7f46840fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:25.193 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.286+0000 7f468af5e700 1 -- 172.21.15.6:0/2328672760 shutdown_connections 2024-02-09T21:50:25.193 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:24.286+0000 7f468af5e700 1 -- 172.21.15.6:0/2328672760 wait complete. 2024-02-09T21:50:25.193 INFO:teuthology.orchestra.run.smithi006.stdout:Deploying grafana service with default placement... 2024-02-09T21:50:25.601 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:25 smithi006 ceph-mon[29258]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:50:25.601 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:25 smithi006 ceph-mon[29258]: Saving service prometheus spec with placement count:1 2024-02-09T21:50:25.602 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:25 smithi006 ceph-mon[29258]: from='mgr.14120 172.21.15.6:0/3931215362' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:26.471 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-02-09T21:50:26.471 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.615+0000 7f9a6d02d700 1 Processor -- start 2024-02-09T21:50:26.471 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.615+0000 7f9a6d02d700 1 -- start start 2024-02-09T21:50:26.471 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.615+0000 7f9a6d02d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a68102770 0x7f9a68102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:26.471 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.615+0000 7f9a6d02d700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a681030c0 con 0x7f9a68102770 2024-02-09T21:50:26.471 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.615+0000 7f9a66d9d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a68102770 0x7f9a68102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:26.471 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.615+0000 7f9a66d9d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a68102770 0x7f9a68102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41064/0 (socket says 172.21.15.6:41064) 2024-02-09T21:50:26.471 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.615+0000 7f9a66d9d700 1 -- 172.21.15.6:0/2718447011 learned_addr learned my addr 172.21.15.6:0/2718447011 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:26.472 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.615+0000 7f9a66d9d700 1 -- 172.21.15.6:0/2718447011 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a68103200 con 0x7f9a68102770 2024-02-09T21:50:26.472 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.615+0000 7f9a66d9d700 1 --2- 172.21.15.6:0/2718447011 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a68102770 0x7f9a68102b80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9a580098e0 tx=0x7f9a58009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98f2f4b889799c8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:26.472 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.616+0000 7f9a65d9b700 1 -- 172.21.15.6:0/2718447011 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a5800b860 con 0x7f9a68102770 2024-02-09T21:50:26.472 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.616+0000 7f9a65d9b700 1 -- 172.21.15.6:0/2718447011 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9a5800be90 con 0x7f9a68102770 2024-02-09T21:50:26.472 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.616+0000 7f9a65d9b700 1 -- 172.21.15.6:0/2718447011 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a580103c0 con 0x7f9a68102770 2024-02-09T21:50:26.472 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.616+0000 7f9a6d02d700 1 -- 172.21.15.6:0/2718447011 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a68102770 msgr2=0x7f9a68102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:26.472 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.616+0000 7f9a6d02d700 1 --2- 172.21.15.6:0/2718447011 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a68102770 0x7f9a68102b80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9a580098e0 tx=0x7f9a58009250 comp rx=0 tx=0).stop 2024-02-09T21:50:26.472 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.616+0000 7f9a6d02d700 1 -- 172.21.15.6:0/2718447011 shutdown_connections 2024-02-09T21:50:26.472 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.616+0000 7f9a6d02d700 1 --2- 172.21.15.6:0/2718447011 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a68102770 0x7f9a68102b80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:26.472 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.616+0000 7f9a6d02d700 1 -- 172.21.15.6:0/2718447011 >> 172.21.15.6:0/2718447011 conn(0x7f9a680fded0 msgr2=0x7f9a68100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:26.472 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.616+0000 7f9a6d02d700 1 -- 172.21.15.6:0/2718447011 shutdown_connections 2024-02-09T21:50:26.472 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.616+0000 7f9a6d02d700 1 -- 172.21.15.6:0/2718447011 wait complete. 2024-02-09T21:50:26.472 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.617+0000 7f9a6d02d700 1 Processor -- start 2024-02-09T21:50:26.472 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.617+0000 7f9a6d02d700 1 -- start start 2024-02-09T21:50:26.472 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.617+0000 7f9a6d02d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a68102770 0x7f9a68078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:26.473 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.617+0000 7f9a6d02d700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a68077190 con 0x7f9a68102770 2024-02-09T21:50:26.473 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.617+0000 7f9a66d9d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a68102770 0x7f9a68078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:26.473 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.617+0000 7f9a66d9d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a68102770 0x7f9a68078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41066/0 (socket says 172.21.15.6:41066) 2024-02-09T21:50:26.473 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.617+0000 7f9a66d9d700 1 -- 172.21.15.6:0/236932382 learned_addr learned my addr 172.21.15.6:0/236932382 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:26.473 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.617+0000 7f9a66d9d700 1 -- 172.21.15.6:0/236932382 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a580094f0 con 0x7f9a68102770 2024-02-09T21:50:26.473 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.618+0000 7f9a66d9d700 1 --2- 172.21.15.6:0/236932382 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a68102770 0x7f9a68078d10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9a580090b0 tx=0x7f9a58009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:26.473 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.618+0000 7f9a4f7fe700 1 -- 172.21.15.6:0/236932382 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a58010840 con 0x7f9a68102770 2024-02-09T21:50:26.473 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.618+0000 7f9a4f7fe700 1 -- 172.21.15.6:0/236932382 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9a58010e70 con 0x7f9a68102770 2024-02-09T21:50:26.473 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.618+0000 7f9a6d02d700 1 -- 172.21.15.6:0/236932382 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a68077390 con 0x7f9a68102770 2024-02-09T21:50:26.473 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.618+0000 7f9a4f7fe700 1 -- 172.21.15.6:0/236932382 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a58024e40 con 0x7f9a68102770 2024-02-09T21:50:26.473 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.618+0000 7f9a6d02d700 1 -- 172.21.15.6:0/236932382 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a68077820 con 0x7f9a68102770 2024-02-09T21:50:26.473 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.619+0000 7f9a4f7fe700 1 -- 172.21.15.6:0/236932382 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f9a5801a520 con 0x7f9a68102770 2024-02-09T21:50:26.473 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.619+0000 7f9a4f7fe700 1 --2- 172.21.15.6:0/236932382 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f9a50036790 0x7f9a50038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:26.473 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.619+0000 7f9a6d02d700 1 -- 172.21.15.6:0/236932382 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a6804bed0 con 0x7f9a68102770 2024-02-09T21:50:26.473 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.619+0000 7f9a4f7fe700 1 -- 172.21.15.6:0/236932382 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f9a5804a7d0 con 0x7f9a68102770 2024-02-09T21:50:26.474 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.619+0000 7f9a6659c700 1 --2- 172.21.15.6:0/236932382 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f9a50036790 0x7f9a50038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:26.474 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.619+0000 7f9a6659c700 1 --2- 172.21.15.6:0/236932382 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f9a50036790 0x7f9a50038c40 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c009990 tx=0x7f9a5c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:26.474 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.623+0000 7f9a4f7fe700 1 -- 172.21.15.6:0/236932382 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9a58018940 con 0x7f9a68102770 2024-02-09T21:50:26.474 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.828+0000 7f9a6d02d700 1 -- 172.21.15.6:0/236932382 --> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a68077fa0 con 0x7f9a50036790 2024-02-09T21:50:26.474 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.832+0000 7f9a4f7fe700 1 -- 172.21.15.6:0/236932382 <== mgr.14120 v2:172.21.15.6:6800/1422219104 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f9a68077fa0 con 0x7f9a50036790 2024-02-09T21:50:26.474 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.834+0000 7f9a6d02d700 1 -- 172.21.15.6:0/236932382 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f9a50036790 msgr2=0x7f9a50038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:26.474 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.834+0000 7f9a6d02d700 1 --2- 172.21.15.6:0/236932382 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f9a50036790 0x7f9a50038c40 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c009990 tx=0x7f9a5c006e40 comp rx=0 tx=0).stop 2024-02-09T21:50:26.474 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.834+0000 7f9a6d02d700 1 -- 172.21.15.6:0/236932382 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a68102770 msgr2=0x7f9a68078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:26.474 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.834+0000 7f9a6d02d700 1 --2- 172.21.15.6:0/236932382 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a68102770 0x7f9a68078d10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9a580090b0 tx=0x7f9a58009250 comp rx=0 tx=0).stop 2024-02-09T21:50:26.474 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.834+0000 7f9a6d02d700 1 -- 172.21.15.6:0/236932382 shutdown_connections 2024-02-09T21:50:26.474 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.834+0000 7f9a6d02d700 1 --2- 172.21.15.6:0/236932382 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f9a50036790 0x7f9a50038c40 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:26.474 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.834+0000 7f9a6d02d700 1 --2- 172.21.15.6:0/236932382 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a68102770 0x7f9a68078d10 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:26.474 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.834+0000 7f9a6d02d700 1 -- 172.21.15.6:0/236932382 >> 172.21.15.6:0/236932382 conn(0x7f9a680fded0 msgr2=0x7f9a680ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:26.474 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.834+0000 7f9a6d02d700 1 -- 172.21.15.6:0/236932382 shutdown_connections 2024-02-09T21:50:26.475 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:25.834+0000 7f9a6d02d700 1 -- 172.21.15.6:0/236932382 wait complete. 2024-02-09T21:50:26.475 INFO:teuthology.orchestra.run.smithi006.stdout:Deploying node-exporter service with default placement... 2024-02-09T21:50:27.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:26 smithi006 ceph-mon[29258]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:50:27.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:26 smithi006 ceph-mon[29258]: Saving service grafana spec with placement count:1 2024-02-09T21:50:27.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:26 smithi006 ceph-mon[29258]: from='mgr.14120 172.21.15.6:0/3931215362' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:27.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:26 smithi006 ceph-mon[29258]: from='mgr.14120 172.21.15.6:0/3931215362' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:28.160 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:28 smithi006 ceph-mon[29258]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:50:28.160 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:28 smithi006 ceph-mon[29258]: Saving service node-exporter spec with placement * 2024-02-09T21:50:28.160 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:28 smithi006 ceph-mon[29258]: from='mgr.14120 172.21.15.6:0/3931215362' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:28.502 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-02-09T21:50:28.502 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.867+0000 7f37872cc700 1 Processor -- start 2024-02-09T21:50:28.502 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.867+0000 7f37872cc700 1 -- start start 2024-02-09T21:50:28.502 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.867+0000 7f37872cc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3780101670 0x7f3780101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:28.502 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.867+0000 7f37872cc700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3780101fc0 con 0x7f3780101670 2024-02-09T21:50:28.502 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.868+0000 7f3785068700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3780101670 0x7f3780101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:28.502 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.868+0000 7f3785068700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3780101670 0x7f3780101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41080/0 (socket says 172.21.15.6:41080) 2024-02-09T21:50:28.502 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.868+0000 7f3785068700 1 -- 172.21.15.6:0/3865736974 learned_addr learned my addr 172.21.15.6:0/3865736974 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:28.502 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.868+0000 7f3785068700 1 -- 172.21.15.6:0/3865736974 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3780102100 con 0x7f3780101670 2024-02-09T21:50:28.502 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.868+0000 7f3785068700 1 --2- 172.21.15.6:0/3865736974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3780101670 0x7f3780101a80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f377400bb80 tx=0x7f377400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d891622eff47dd38 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:28.502 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.868+0000 7f3773fff700 1 -- 172.21.15.6:0/3865736974 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f377400dd00 con 0x7f3780101670 2024-02-09T21:50:28.503 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.868+0000 7f3773fff700 1 -- 172.21.15.6:0/3865736974 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f377400de60 con 0x7f3780101670 2024-02-09T21:50:28.503 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.868+0000 7f3773fff700 1 -- 172.21.15.6:0/3865736974 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3774018920 con 0x7f3780101670 2024-02-09T21:50:28.503 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.869+0000 7f37872cc700 1 -- 172.21.15.6:0/3865736974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3780101670 msgr2=0x7f3780101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:28.503 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.869+0000 7f37872cc700 1 --2- 172.21.15.6:0/3865736974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3780101670 0x7f3780101a80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f377400bb80 tx=0x7f377400b4f0 comp rx=0 tx=0).stop 2024-02-09T21:50:28.503 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.869+0000 7f37872cc700 1 -- 172.21.15.6:0/3865736974 shutdown_connections 2024-02-09T21:50:28.503 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.869+0000 7f37872cc700 1 --2- 172.21.15.6:0/3865736974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3780101670 0x7f3780101a80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:28.503 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.869+0000 7f37872cc700 1 -- 172.21.15.6:0/3865736974 >> 172.21.15.6:0/3865736974 conn(0x7f37800fcdd0 msgr2=0x7f37800ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:28.503 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.869+0000 7f37872cc700 1 -- 172.21.15.6:0/3865736974 shutdown_connections 2024-02-09T21:50:28.503 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.869+0000 7f37872cc700 1 -- 172.21.15.6:0/3865736974 wait complete. 2024-02-09T21:50:28.503 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.870+0000 7f37872cc700 1 Processor -- start 2024-02-09T21:50:28.503 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.870+0000 7f37872cc700 1 -- start start 2024-02-09T21:50:28.503 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.870+0000 7f37872cc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3780101670 0x7f378010f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:28.503 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.870+0000 7f37872cc700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378010f880 con 0x7f3780101670 2024-02-09T21:50:28.503 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.870+0000 7f3785068700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3780101670 0x7f378010f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:28.504 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.870+0000 7f3785068700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3780101670 0x7f378010f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41092/0 (socket says 172.21.15.6:41092) 2024-02-09T21:50:28.504 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.870+0000 7f3785068700 1 -- 172.21.15.6:0/1060227333 learned_addr learned my addr 172.21.15.6:0/1060227333 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:28.504 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.871+0000 7f3785068700 1 -- 172.21.15.6:0/1060227333 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f377400b790 con 0x7f3780101670 2024-02-09T21:50:28.504 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.871+0000 7f3785068700 1 --2- 172.21.15.6:0/1060227333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3780101670 0x7f378010f340 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f37740106a0 tx=0x7f377400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:28.504 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.871+0000 7f3771ffb700 1 -- 172.21.15.6:0/1060227333 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3774010bf0 con 0x7f3780101670 2024-02-09T21:50:28.504 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.871+0000 7f37872cc700 1 -- 172.21.15.6:0/1060227333 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f378010fa80 con 0x7f3780101670 2024-02-09T21:50:28.504 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.871+0000 7f3771ffb700 1 -- 172.21.15.6:0/1060227333 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f377401d050 con 0x7f3780101670 2024-02-09T21:50:28.504 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.871+0000 7f3771ffb700 1 -- 172.21.15.6:0/1060227333 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3774022620 con 0x7f3780101670 2024-02-09T21:50:28.504 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.871+0000 7f37872cc700 1 -- 172.21.15.6:0/1060227333 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3780110720 con 0x7f3780101670 2024-02-09T21:50:28.504 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.872+0000 7f3771ffb700 1 -- 172.21.15.6:0/1060227333 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f3774022780 con 0x7f3780101670 2024-02-09T21:50:28.504 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.872+0000 7f37872cc700 1 -- 172.21.15.6:0/1060227333 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f378005f5c0 con 0x7f3780101670 2024-02-09T21:50:28.505 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.872+0000 7f3771ffb700 1 --2- 172.21.15.6:0/1060227333 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f376c0367e0 0x7f376c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:28.505 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.873+0000 7f3771ffb700 1 -- 172.21.15.6:0/1060227333 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f3774014070 con 0x7f3780101670 2024-02-09T21:50:28.505 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.873+0000 7f3784867700 1 --2- 172.21.15.6:0/1060227333 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f376c0367e0 0x7f376c038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:28.505 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.873+0000 7f3784867700 1 --2- 172.21.15.6:0/1060227333 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f376c0367e0 0x7f376c038c90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f377c009990 tx=0x7f377c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:28.505 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:26.878+0000 7f3771ffb700 1 -- 172.21.15.6:0/1060227333 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f377400e360 con 0x7f3780101670 2024-02-09T21:50:28.505 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:27.086+0000 7f37872cc700 1 -- 172.21.15.6:0/1060227333 --> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f3780111300 con 0x7f376c0367e0 2024-02-09T21:50:28.505 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:27.091+0000 7f3771ffb700 1 -- 172.21.15.6:0/1060227333 <== mgr.14120 v2:172.21.15.6:6800/1422219104 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f3780111300 con 0x7f376c0367e0 2024-02-09T21:50:28.505 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:27.094+0000 7f37872cc700 1 -- 172.21.15.6:0/1060227333 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f376c0367e0 msgr2=0x7f376c038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:28.505 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:27.094+0000 7f37872cc700 1 --2- 172.21.15.6:0/1060227333 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f376c0367e0 0x7f376c038c90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f377c009990 tx=0x7f377c006e40 comp rx=0 tx=0).stop 2024-02-09T21:50:28.505 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:27.094+0000 7f37872cc700 1 -- 172.21.15.6:0/1060227333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3780101670 msgr2=0x7f378010f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:28.505 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:27.094+0000 7f37872cc700 1 --2- 172.21.15.6:0/1060227333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3780101670 0x7f378010f340 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f37740106a0 tx=0x7f377400b4f0 comp rx=0 tx=0).stop 2024-02-09T21:50:28.505 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:27.094+0000 7f37872cc700 1 -- 172.21.15.6:0/1060227333 shutdown_connections 2024-02-09T21:50:28.505 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:27.094+0000 7f37872cc700 1 --2- 172.21.15.6:0/1060227333 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f376c0367e0 0x7f376c038c90 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:28.505 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:27.094+0000 7f37872cc700 1 --2- 172.21.15.6:0/1060227333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3780101670 0x7f378010f340 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:28.506 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:27.094+0000 7f37872cc700 1 -- 172.21.15.6:0/1060227333 >> 172.21.15.6:0/1060227333 conn(0x7f37800fcdd0 msgr2=0x7f37800ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:28.506 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:27.094+0000 7f37872cc700 1 -- 172.21.15.6:0/1060227333 shutdown_connections 2024-02-09T21:50:28.506 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:27.094+0000 7f37872cc700 1 -- 172.21.15.6:0/1060227333 wait complete. 2024-02-09T21:50:28.506 INFO:teuthology.orchestra.run.smithi006.stdout:Deploying alertmanager service with default placement... 2024-02-09T21:50:29.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:29 smithi006 ceph-mon[29258]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:50:29.750 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-02-09T21:50:29.750 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.922+0000 7fb368d52700 1 Processor -- start 2024-02-09T21:50:29.750 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.922+0000 7fb368d52700 1 -- start start 2024-02-09T21:50:29.750 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.922+0000 7fb368d52700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb364102770 0x7fb364102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:29.750 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.922+0000 7fb368d52700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3641030c0 con 0x7fb364102770 2024-02-09T21:50:29.750 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.922+0000 7fb36259c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb364102770 0x7fb364102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:29.750 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.922+0000 7fb36259c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb364102770 0x7fb364102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60000/0 (socket says 172.21.15.6:60000) 2024-02-09T21:50:29.751 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.922+0000 7fb36259c700 1 -- 172.21.15.6:0/3964212944 learned_addr learned my addr 172.21.15.6:0/3964212944 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:29.751 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.923+0000 7fb36259c700 1 -- 172.21.15.6:0/3964212944 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb364103200 con 0x7fb364102770 2024-02-09T21:50:29.751 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.923+0000 7fb36259c700 1 --2- 172.21.15.6:0/3964212944 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb364102770 0x7fb364102b80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb3540098e0 tx=0x7fb354009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87030c31962130d7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:29.751 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.923+0000 7fb36159a700 1 -- 172.21.15.6:0/3964212944 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb35400b860 con 0x7fb364102770 2024-02-09T21:50:29.751 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.923+0000 7fb36159a700 1 -- 172.21.15.6:0/3964212944 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb35400be90 con 0x7fb364102770 2024-02-09T21:50:29.751 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.923+0000 7fb36159a700 1 -- 172.21.15.6:0/3964212944 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3540103c0 con 0x7fb364102770 2024-02-09T21:50:29.751 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.924+0000 7fb368d52700 1 -- 172.21.15.6:0/3964212944 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb364102770 msgr2=0x7fb364102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:29.751 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.924+0000 7fb368d52700 1 --2- 172.21.15.6:0/3964212944 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb364102770 0x7fb364102b80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb3540098e0 tx=0x7fb354009250 comp rx=0 tx=0).stop 2024-02-09T21:50:29.751 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.924+0000 7fb368d52700 1 -- 172.21.15.6:0/3964212944 shutdown_connections 2024-02-09T21:50:29.751 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.924+0000 7fb368d52700 1 --2- 172.21.15.6:0/3964212944 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb364102770 0x7fb364102b80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:29.751 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.924+0000 7fb368d52700 1 -- 172.21.15.6:0/3964212944 >> 172.21.15.6:0/3964212944 conn(0x7fb3640fded0 msgr2=0x7fb364100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:29.752 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.924+0000 7fb368d52700 1 -- 172.21.15.6:0/3964212944 shutdown_connections 2024-02-09T21:50:29.752 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.924+0000 7fb368d52700 1 -- 172.21.15.6:0/3964212944 wait complete. 2024-02-09T21:50:29.752 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.924+0000 7fb368d52700 1 Processor -- start 2024-02-09T21:50:29.752 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.924+0000 7fb368d52700 1 -- start start 2024-02-09T21:50:29.752 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.924+0000 7fb368d52700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb364102770 0x7fb3641115b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:29.752 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.924+0000 7fb368d52700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb364111af0 con 0x7fb364102770 2024-02-09T21:50:29.752 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.925+0000 7fb36259c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb364102770 0x7fb3641115b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:29.752 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.925+0000 7fb36259c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb364102770 0x7fb3641115b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60008/0 (socket says 172.21.15.6:60008) 2024-02-09T21:50:29.753 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.925+0000 7fb36259c700 1 -- 172.21.15.6:0/3294294131 learned_addr learned my addr 172.21.15.6:0/3294294131 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:29.753 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.925+0000 7fb36259c700 1 -- 172.21.15.6:0/3294294131 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3540094f0 con 0x7fb364102770 2024-02-09T21:50:29.753 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.925+0000 7fb36259c700 1 --2- 172.21.15.6:0/3294294131 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb364102770 0x7fb3641115b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb3540090b0 tx=0x7fb354009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:29.753 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.925+0000 7fb352ffd700 1 -- 172.21.15.6:0/3294294131 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb354010840 con 0x7fb364102770 2024-02-09T21:50:29.753 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.925+0000 7fb352ffd700 1 -- 172.21.15.6:0/3294294131 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb354010e70 con 0x7fb364102770 2024-02-09T21:50:29.753 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.925+0000 7fb368d52700 1 -- 172.21.15.6:0/3294294131 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb364111cf0 con 0x7fb364102770 2024-02-09T21:50:29.753 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.925+0000 7fb352ffd700 1 -- 172.21.15.6:0/3294294131 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb354024e40 con 0x7fb364102770 2024-02-09T21:50:29.753 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.925+0000 7fb368d52700 1 -- 172.21.15.6:0/3294294131 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb364112990 con 0x7fb364102770 2024-02-09T21:50:29.753 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.926+0000 7fb352ffd700 1 -- 172.21.15.6:0/3294294131 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fb35401a4c0 con 0x7fb364102770 2024-02-09T21:50:29.753 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.926+0000 7fb352ffd700 1 --2- 172.21.15.6:0/3294294131 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fb34c036790 0x7fb34c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:29.753 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.926+0000 7fb352ffd700 1 -- 172.21.15.6:0/3294294131 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb35404a790 con 0x7fb364102770 2024-02-09T21:50:29.754 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.926+0000 7fb368d52700 1 -- 172.21.15.6:0/3294294131 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb36405f5c0 con 0x7fb364102770 2024-02-09T21:50:29.754 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.926+0000 7fb361d9b700 1 --2- 172.21.15.6:0/3294294131 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fb34c036790 0x7fb34c038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:29.754 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.927+0000 7fb361d9b700 1 --2- 172.21.15.6:0/3294294131 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fb34c036790 0x7fb34c038c40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb358009990 tx=0x7fb358006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:29.754 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:28.931+0000 7fb352ffd700 1 -- 172.21.15.6:0/3294294131 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb354018940 con 0x7fb364102770 2024-02-09T21:50:29.754 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:29.137+0000 7fb368d52700 1 -- 172.21.15.6:0/3294294131 --> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3641001b0 con 0x7fb34c036790 2024-02-09T21:50:29.754 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:29.141+0000 7fb352ffd700 1 -- 172.21.15.6:0/3294294131 <== mgr.14120 v2:172.21.15.6:6800/1422219104 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fb3641001b0 con 0x7fb34c036790 2024-02-09T21:50:29.754 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:29.143+0000 7fb368d52700 1 -- 172.21.15.6:0/3294294131 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fb34c036790 msgr2=0x7fb34c038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:29.754 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:29.143+0000 7fb368d52700 1 --2- 172.21.15.6:0/3294294131 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fb34c036790 0x7fb34c038c40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb358009990 tx=0x7fb358006e40 comp rx=0 tx=0).stop 2024-02-09T21:50:29.754 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:29.143+0000 7fb368d52700 1 -- 172.21.15.6:0/3294294131 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb364102770 msgr2=0x7fb3641115b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:29.754 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:29.143+0000 7fb368d52700 1 --2- 172.21.15.6:0/3294294131 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb364102770 0x7fb3641115b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb3540090b0 tx=0x7fb354009250 comp rx=0 tx=0).stop 2024-02-09T21:50:29.754 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:29.143+0000 7fb368d52700 1 -- 172.21.15.6:0/3294294131 shutdown_connections 2024-02-09T21:50:29.754 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:29.143+0000 7fb368d52700 1 --2- 172.21.15.6:0/3294294131 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fb34c036790 0x7fb34c038c40 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:29.754 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:29.143+0000 7fb368d52700 1 --2- 172.21.15.6:0/3294294131 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb364102770 0x7fb3641115b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:29.754 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:29.143+0000 7fb368d52700 1 -- 172.21.15.6:0/3294294131 >> 172.21.15.6:0/3294294131 conn(0x7fb3640fded0 msgr2=0x7fb3640ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:29.755 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:29.143+0000 7fb368d52700 1 -- 172.21.15.6:0/3294294131 shutdown_connections 2024-02-09T21:50:29.755 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:29.143+0000 7fb368d52700 1 -- 172.21.15.6:0/3294294131 wait complete. 2024-02-09T21:50:30.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:30 smithi006 ceph-mon[29258]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:50:30.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:30 smithi006 ceph-mon[29258]: Saving service alertmanager spec with placement count:1 2024-02-09T21:50:30.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:30 smithi006 ceph-mon[29258]: from='mgr.14120 172.21.15.6:0/3931215362' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:30.973 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.130+0000 7f1dc464a700 1 Processor -- start 2024-02-09T21:50:30.973 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.130+0000 7f1dc464a700 1 -- start start 2024-02-09T21:50:30.973 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.130+0000 7f1dc464a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1dbc101670 0x7f1dbc101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:30.973 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.130+0000 7f1dc464a700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dbc101fc0 con 0x7f1dbc101670 2024-02-09T21:50:30.973 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.130+0000 7f1dc23e6700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1dbc101670 0x7f1dbc101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:30.973 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.131+0000 7f1dc23e6700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1dbc101670 0x7f1dbc101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60018/0 (socket says 172.21.15.6:60018) 2024-02-09T21:50:30.973 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.131+0000 7f1dc23e6700 1 -- 172.21.15.6:0/1485505985 learned_addr learned my addr 172.21.15.6:0/1485505985 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:30.973 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.131+0000 7f1dc23e6700 1 -- 172.21.15.6:0/1485505985 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dbc102100 con 0x7f1dbc101670 2024-02-09T21:50:30.973 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.131+0000 7f1dc23e6700 1 --2- 172.21.15.6:0/1485505985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1dbc101670 0x7f1dbc101a80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1db40098e0 tx=0x7f1db4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a10a16cfb69978db server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:30.974 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.131+0000 7f1dc13e4700 1 -- 172.21.15.6:0/1485505985 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1db400b860 con 0x7f1dbc101670 2024-02-09T21:50:30.974 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.131+0000 7f1dc13e4700 1 -- 172.21.15.6:0/1485505985 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1db400be90 con 0x7f1dbc101670 2024-02-09T21:50:30.974 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.131+0000 7f1dc13e4700 1 -- 172.21.15.6:0/1485505985 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1db40103c0 con 0x7f1dbc101670 2024-02-09T21:50:30.974 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.132+0000 7f1dc464a700 1 -- 172.21.15.6:0/1485505985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1dbc101670 msgr2=0x7f1dbc101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:30.974 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.132+0000 7f1dc464a700 1 --2- 172.21.15.6:0/1485505985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1dbc101670 0x7f1dbc101a80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1db40098e0 tx=0x7f1db4009250 comp rx=0 tx=0).stop 2024-02-09T21:50:30.974 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.132+0000 7f1dc464a700 1 -- 172.21.15.6:0/1485505985 shutdown_connections 2024-02-09T21:50:30.974 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.132+0000 7f1dc464a700 1 --2- 172.21.15.6:0/1485505985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1dbc101670 0x7f1dbc101a80 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:30.974 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.132+0000 7f1dc464a700 1 -- 172.21.15.6:0/1485505985 >> 172.21.15.6:0/1485505985 conn(0x7f1dbc0fcdd0 msgr2=0x7f1dbc0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:30.974 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.132+0000 7f1dc464a700 1 -- 172.21.15.6:0/1485505985 shutdown_connections 2024-02-09T21:50:30.974 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.132+0000 7f1dc464a700 1 -- 172.21.15.6:0/1485505985 wait complete. 2024-02-09T21:50:30.974 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.132+0000 7f1dc464a700 1 Processor -- start 2024-02-09T21:50:30.974 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.133+0000 7f1dc464a700 1 -- start start 2024-02-09T21:50:30.974 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.133+0000 7f1dc464a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1dbc101670 0x7f1dbc111850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:30.974 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.133+0000 7f1dc464a700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dbc111d90 con 0x7f1dbc101670 2024-02-09T21:50:30.975 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.133+0000 7f1dc23e6700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1dbc101670 0x7f1dbc111850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:30.975 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.133+0000 7f1dc23e6700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1dbc101670 0x7f1dbc111850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60020/0 (socket says 172.21.15.6:60020) 2024-02-09T21:50:30.975 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.133+0000 7f1dc23e6700 1 -- 172.21.15.6:0/3571200626 learned_addr learned my addr 172.21.15.6:0/3571200626 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:30.975 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.133+0000 7f1dc23e6700 1 -- 172.21.15.6:0/3571200626 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db40094f0 con 0x7f1dbc101670 2024-02-09T21:50:30.975 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.133+0000 7f1dc23e6700 1 --2- 172.21.15.6:0/3571200626 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1dbc101670 0x7f1dbc111850 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1db4006b00 tx=0x7f1db4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:30.975 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.133+0000 7f1dbaffd700 1 -- 172.21.15.6:0/3571200626 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1db40109a0 con 0x7f1dbc101670 2024-02-09T21:50:30.975 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.133+0000 7f1dbaffd700 1 -- 172.21.15.6:0/3571200626 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1db401a430 con 0x7f1dbc101670 2024-02-09T21:50:30.975 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.133+0000 7f1dc464a700 1 -- 172.21.15.6:0/3571200626 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dbc111f90 con 0x7f1dbc101670 2024-02-09T21:50:30.975 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.133+0000 7f1dbaffd700 1 -- 172.21.15.6:0/3571200626 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1db40245b0 con 0x7f1dbc101670 2024-02-09T21:50:30.975 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.133+0000 7f1dc464a700 1 -- 172.21.15.6:0/3571200626 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dbc112c30 con 0x7f1dbc101670 2024-02-09T21:50:30.975 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.134+0000 7f1dbaffd700 1 -- 172.21.15.6:0/3571200626 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f1db4010b00 con 0x7f1dbc101670 2024-02-09T21:50:30.975 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.134+0000 7f1dbaffd700 1 --2- 172.21.15.6:0/3571200626 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f1da40367e0 0x7f1da4038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:30.975 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.134+0000 7f1dc464a700 1 -- 172.21.15.6:0/3571200626 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dbc05f5c0 con 0x7f1dbc101670 2024-02-09T21:50:30.975 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.135+0000 7f1dbaffd700 1 -- 172.21.15.6:0/3571200626 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f1db404a640 con 0x7f1dbc101670 2024-02-09T21:50:30.976 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.135+0000 7f1dc1be5700 1 --2- 172.21.15.6:0/3571200626 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f1da40367e0 0x7f1da4038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:30.976 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.135+0000 7f1dc1be5700 1 --2- 172.21.15.6:0/3571200626 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f1da40367e0 0x7f1da4038c90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1db0009990 tx=0x7f1db0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:30.976 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.139+0000 7f1dbaffd700 1 -- 172.21.15.6:0/3571200626 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1db401aaa0 con 0x7f1dbc101670 2024-02-09T21:50:30.976 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.331+0000 7f1dc464a700 1 -- 172.21.15.6:0/3571200626 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f1dbc1035b0 con 0x7f1dbc101670 2024-02-09T21:50:30.976 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.335+0000 7f1dbaffd700 1 -- 172.21.15.6:0/3571200626 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f1db4018b40 con 0x7f1dbc101670 2024-02-09T21:50:30.976 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.337+0000 7f1dc464a700 1 -- 172.21.15.6:0/3571200626 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f1da40367e0 msgr2=0x7f1da4038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:30.976 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.337+0000 7f1dc464a700 1 --2- 172.21.15.6:0/3571200626 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f1da40367e0 0x7f1da4038c90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1db0009990 tx=0x7f1db0006e40 comp rx=0 tx=0).stop 2024-02-09T21:50:30.976 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.337+0000 7f1dc464a700 1 -- 172.21.15.6:0/3571200626 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1dbc101670 msgr2=0x7f1dbc111850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:30.976 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.337+0000 7f1dc464a700 1 --2- 172.21.15.6:0/3571200626 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1dbc101670 0x7f1dbc111850 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1db4006b00 tx=0x7f1db4009250 comp rx=0 tx=0).stop 2024-02-09T21:50:30.976 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.337+0000 7f1dc464a700 1 -- 172.21.15.6:0/3571200626 shutdown_connections 2024-02-09T21:50:30.976 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.337+0000 7f1dc464a700 1 --2- 172.21.15.6:0/3571200626 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f1da40367e0 0x7f1da4038c90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:30.976 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.337+0000 7f1dc464a700 1 --2- 172.21.15.6:0/3571200626 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1dbc101670 0x7f1dbc111850 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:30.976 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.338+0000 7f1dc464a700 1 -- 172.21.15.6:0/3571200626 >> 172.21.15.6:0/3571200626 conn(0x7f1dbc0fcdd0 msgr2=0x7f1dbc0fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:30.976 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.338+0000 7f1dc464a700 1 -- 172.21.15.6:0/3571200626 shutdown_connections 2024-02-09T21:50:30.976 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:30.338+0000 7f1dc464a700 1 -- 172.21.15.6:0/3571200626 wait complete. 2024-02-09T21:50:31.586 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:31 smithi006 ceph-mon[29258]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:50:31.586 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:31 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/3571200626' entity='client.admin' 2024-02-09T21:50:32.187 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.380+0000 7f86ee331700 1 Processor -- start 2024-02-09T21:50:32.188 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.381+0000 7f86ee331700 1 -- start start 2024-02-09T21:50:32.188 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.381+0000 7f86ee331700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86e8076c60 0x7f86e8105340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:32.188 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.381+0000 7f86ee331700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e8105880 con 0x7f86e8076c60 2024-02-09T21:50:32.188 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.381+0000 7f86e7fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86e8076c60 0x7f86e8105340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:32.188 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.381+0000 7f86e7fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86e8076c60 0x7f86e8105340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60022/0 (socket says 172.21.15.6:60022) 2024-02-09T21:50:32.188 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.381+0000 7f86e7fff700 1 -- 172.21.15.6:0/2587490293 learned_addr learned my addr 172.21.15.6:0/2587490293 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:32.188 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.381+0000 7f86e7fff700 1 -- 172.21.15.6:0/2587490293 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e81059c0 con 0x7f86e8076c60 2024-02-09T21:50:32.188 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.381+0000 7f86e7fff700 1 --2- 172.21.15.6:0/2587490293 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86e8076c60 0x7f86e8105340 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f86d80098e0 tx=0x7f86d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abb2b1b6929a7f90 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:32.188 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.381+0000 7f86e6ffd700 1 -- 172.21.15.6:0/2587490293 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86d800b860 con 0x7f86e8076c60 2024-02-09T21:50:32.188 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.382+0000 7f86e6ffd700 1 -- 172.21.15.6:0/2587490293 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f86d800be90 con 0x7f86e8076c60 2024-02-09T21:50:32.188 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.382+0000 7f86e6ffd700 1 -- 172.21.15.6:0/2587490293 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86d80103c0 con 0x7f86e8076c60 2024-02-09T21:50:32.188 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.382+0000 7f86ee331700 1 -- 172.21.15.6:0/2587490293 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86e8076c60 msgr2=0x7f86e8105340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:32.188 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.382+0000 7f86ee331700 1 --2- 172.21.15.6:0/2587490293 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86e8076c60 0x7f86e8105340 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f86d80098e0 tx=0x7f86d8009250 comp rx=0 tx=0).stop 2024-02-09T21:50:32.189 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.382+0000 7f86ee331700 1 -- 172.21.15.6:0/2587490293 shutdown_connections 2024-02-09T21:50:32.189 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.382+0000 7f86ee331700 1 --2- 172.21.15.6:0/2587490293 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86e8076c60 0x7f86e8105340 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:32.189 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.382+0000 7f86ee331700 1 -- 172.21.15.6:0/2587490293 >> 172.21.15.6:0/2587490293 conn(0x7f86e80fcdd0 msgr2=0x7f86e80ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:32.189 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.382+0000 7f86ee331700 1 -- 172.21.15.6:0/2587490293 shutdown_connections 2024-02-09T21:50:32.189 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.382+0000 7f86ee331700 1 -- 172.21.15.6:0/2587490293 wait complete. 2024-02-09T21:50:32.189 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.383+0000 7f86ee331700 1 Processor -- start 2024-02-09T21:50:32.189 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.383+0000 7f86ee331700 1 -- start start 2024-02-09T21:50:32.189 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.383+0000 7f86ee331700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86e8076c60 0x7f86e8102dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:32.189 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.383+0000 7f86ee331700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e8101240 con 0x7f86e8076c60 2024-02-09T21:50:32.189 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.383+0000 7f86e7fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86e8076c60 0x7f86e8102dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:32.189 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.383+0000 7f86e7fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86e8076c60 0x7f86e8102dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60026/0 (socket says 172.21.15.6:60026) 2024-02-09T21:50:32.189 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.383+0000 7f86e7fff700 1 -- 172.21.15.6:0/2946293207 learned_addr learned my addr 172.21.15.6:0/2946293207 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:32.189 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.383+0000 7f86e7fff700 1 -- 172.21.15.6:0/2946293207 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d80094f0 con 0x7f86e8076c60 2024-02-09T21:50:32.189 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.384+0000 7f86e7fff700 1 --2- 172.21.15.6:0/2946293207 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86e8076c60 0x7f86e8102dc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f86d8003080 tx=0x7f86d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:32.189 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.384+0000 7f86e4ff9700 1 -- 172.21.15.6:0/2946293207 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86d8011920 con 0x7f86e8076c60 2024-02-09T21:50:32.190 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.384+0000 7f86ee331700 1 -- 172.21.15.6:0/2946293207 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86e8101440 con 0x7f86e8076c60 2024-02-09T21:50:32.190 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.384+0000 7f86e4ff9700 1 -- 172.21.15.6:0/2946293207 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f86d8010d60 con 0x7f86e8076c60 2024-02-09T21:50:32.190 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.384+0000 7f86e4ff9700 1 -- 172.21.15.6:0/2946293207 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86d80243f0 con 0x7f86e8076c60 2024-02-09T21:50:32.190 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.384+0000 7f86ee331700 1 -- 172.21.15.6:0/2946293207 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86e81018d0 con 0x7f86e8076c60 2024-02-09T21:50:32.190 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.385+0000 7f86e4ff9700 1 -- 172.21.15.6:0/2946293207 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f86d802d3f0 con 0x7f86e8076c60 2024-02-09T21:50:32.190 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.385+0000 7f86ee331700 1 -- 172.21.15.6:0/2946293207 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86e804bed0 con 0x7f86e8076c60 2024-02-09T21:50:32.190 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.385+0000 7f86e4ff9700 1 --2- 172.21.15.6:0/2946293207 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f86d0036790 0x7f86d0038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:32.190 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.385+0000 7f86e4ff9700 1 -- 172.21.15.6:0/2946293207 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f86d804a4b0 con 0x7f86e8076c60 2024-02-09T21:50:32.190 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.385+0000 7f86e77fe700 1 --2- 172.21.15.6:0/2946293207 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f86d0036790 0x7f86d0038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:32.190 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.385+0000 7f86e77fe700 1 --2- 172.21.15.6:0/2946293207 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f86d0036790 0x7f86d0038c40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f86dc009990 tx=0x7f86dc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:32.190 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.390+0000 7f86e4ff9700 1 -- 172.21.15.6:0/2946293207 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f86d802b8e0 con 0x7f86e8076c60 2024-02-09T21:50:32.190 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.582+0000 7f86ee331700 1 -- 172.21.15.6:0/2946293207 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f86e8101e10 con 0x7f86e8076c60 2024-02-09T21:50:32.190 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.586+0000 7f86e4ff9700 1 -- 172.21.15.6:0/2946293207 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f86d8009350 con 0x7f86e8076c60 2024-02-09T21:50:32.191 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.588+0000 7f86ee331700 1 -- 172.21.15.6:0/2946293207 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f86d0036790 msgr2=0x7f86d0038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:32.191 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.588+0000 7f86ee331700 1 --2- 172.21.15.6:0/2946293207 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f86d0036790 0x7f86d0038c40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f86dc009990 tx=0x7f86dc006e40 comp rx=0 tx=0).stop 2024-02-09T21:50:32.191 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.588+0000 7f86ee331700 1 -- 172.21.15.6:0/2946293207 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86e8076c60 msgr2=0x7f86e8102dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:32.191 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.588+0000 7f86ee331700 1 --2- 172.21.15.6:0/2946293207 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86e8076c60 0x7f86e8102dc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f86d8003080 tx=0x7f86d8009250 comp rx=0 tx=0).stop 2024-02-09T21:50:32.191 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.588+0000 7f86ee331700 1 -- 172.21.15.6:0/2946293207 shutdown_connections 2024-02-09T21:50:32.191 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.588+0000 7f86ee331700 1 --2- 172.21.15.6:0/2946293207 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f86d0036790 0x7f86d0038c40 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:32.191 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.588+0000 7f86ee331700 1 --2- 172.21.15.6:0/2946293207 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86e8076c60 0x7f86e8102dc0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:32.191 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.588+0000 7f86ee331700 1 -- 172.21.15.6:0/2946293207 >> 172.21.15.6:0/2946293207 conn(0x7f86e80fcdd0 msgr2=0x7f86e80fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:32.191 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.588+0000 7f86ee331700 1 -- 172.21.15.6:0/2946293207 shutdown_connections 2024-02-09T21:50:32.191 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:31.588+0000 7f86ee331700 1 -- 172.21.15.6:0/2946293207 wait complete. 2024-02-09T21:50:32.191 INFO:teuthology.orchestra.run.smithi006.stdout:Enabling the dashboard module... 2024-02-09T21:50:32.891 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:32 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/2946293207' entity='client.admin' 2024-02-09T21:50:32.891 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:32 smithi006 ceph-mon[29258]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:50:33.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:33 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/351712582' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-09T21:50:34.211 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.629+0000 7fc58e922700 1 Processor -- start 2024-02-09T21:50:34.211 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.629+0000 7fc58e922700 1 -- start start 2024-02-09T21:50:34.211 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.629+0000 7fc58e922700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc588102770 0x7fc588102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:34.212 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.629+0000 7fc58e922700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5881030c0 con 0x7fc588102770 2024-02-09T21:50:34.212 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.629+0000 7fc587fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc588102770 0x7fc588102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:34.212 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.629+0000 7fc587fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc588102770 0x7fc588102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60040/0 (socket says 172.21.15.6:60040) 2024-02-09T21:50:34.212 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.629+0000 7fc587fff700 1 -- 172.21.15.6:0/3549975320 learned_addr learned my addr 172.21.15.6:0/3549975320 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:34.212 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.630+0000 7fc587fff700 1 -- 172.21.15.6:0/3549975320 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc588103200 con 0x7fc588102770 2024-02-09T21:50:34.212 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.630+0000 7fc587fff700 1 --2- 172.21.15.6:0/3549975320 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc588102770 0x7fc588102b80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc5780098e0 tx=0x7fc578009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c302adc4bdb520d2 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:34.212 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.630+0000 7fc586ffd700 1 -- 172.21.15.6:0/3549975320 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc57800b860 con 0x7fc588102770 2024-02-09T21:50:34.212 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.630+0000 7fc586ffd700 1 -- 172.21.15.6:0/3549975320 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc57800be90 con 0x7fc588102770 2024-02-09T21:50:34.212 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.630+0000 7fc586ffd700 1 -- 172.21.15.6:0/3549975320 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5780103c0 con 0x7fc588102770 2024-02-09T21:50:34.212 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.630+0000 7fc58e922700 1 -- 172.21.15.6:0/3549975320 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc588102770 msgr2=0x7fc588102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:34.212 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.630+0000 7fc58e922700 1 --2- 172.21.15.6:0/3549975320 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc588102770 0x7fc588102b80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc5780098e0 tx=0x7fc578009250 comp rx=0 tx=0).stop 2024-02-09T21:50:34.213 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.630+0000 7fc58e922700 1 -- 172.21.15.6:0/3549975320 shutdown_connections 2024-02-09T21:50:34.213 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.630+0000 7fc58e922700 1 --2- 172.21.15.6:0/3549975320 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc588102770 0x7fc588102b80 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:34.213 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.630+0000 7fc58e922700 1 -- 172.21.15.6:0/3549975320 >> 172.21.15.6:0/3549975320 conn(0x7fc5880fded0 msgr2=0x7fc588100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:34.213 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.631+0000 7fc58e922700 1 -- 172.21.15.6:0/3549975320 shutdown_connections 2024-02-09T21:50:34.213 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.631+0000 7fc58e922700 1 -- 172.21.15.6:0/3549975320 wait complete. 2024-02-09T21:50:34.213 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.631+0000 7fc58e922700 1 Processor -- start 2024-02-09T21:50:34.213 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.631+0000 7fc58e922700 1 -- start start 2024-02-09T21:50:34.213 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.631+0000 7fc58e922700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc588102770 0x7fc588078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:34.213 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.631+0000 7fc58e922700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc588077190 con 0x7fc588102770 2024-02-09T21:50:34.213 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.631+0000 7fc587fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc588102770 0x7fc588078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:34.213 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.632+0000 7fc587fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc588102770 0x7fc588078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60044/0 (socket says 172.21.15.6:60044) 2024-02-09T21:50:34.213 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.632+0000 7fc587fff700 1 -- 172.21.15.6:0/351712582 learned_addr learned my addr 172.21.15.6:0/351712582 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:34.213 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.632+0000 7fc587fff700 1 -- 172.21.15.6:0/351712582 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5780094f0 con 0x7fc588102770 2024-02-09T21:50:34.214 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.632+0000 7fc587fff700 1 --2- 172.21.15.6:0/351712582 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc588102770 0x7fc588078d10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc5780090b0 tx=0x7fc578009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:34.214 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.632+0000 7fc584ff9700 1 -- 172.21.15.6:0/351712582 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc578010840 con 0x7fc588102770 2024-02-09T21:50:34.214 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.632+0000 7fc584ff9700 1 -- 172.21.15.6:0/351712582 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc578010e70 con 0x7fc588102770 2024-02-09T21:50:34.214 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.632+0000 7fc58e922700 1 -- 172.21.15.6:0/351712582 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc588077390 con 0x7fc588102770 2024-02-09T21:50:34.214 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.632+0000 7fc584ff9700 1 -- 172.21.15.6:0/351712582 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc578024e40 con 0x7fc588102770 2024-02-09T21:50:34.214 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.632+0000 7fc58e922700 1 -- 172.21.15.6:0/351712582 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc588077820 con 0x7fc588102770 2024-02-09T21:50:34.214 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.633+0000 7fc584ff9700 1 -- 172.21.15.6:0/351712582 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fc57801a4c0 con 0x7fc588102770 2024-02-09T21:50:34.214 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.633+0000 7fc58e922700 1 -- 172.21.15.6:0/351712582 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc58804bed0 con 0x7fc588102770 2024-02-09T21:50:34.214 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.633+0000 7fc584ff9700 1 --2- 172.21.15.6:0/351712582 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fc570036790 0x7fc570038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:34.214 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.633+0000 7fc584ff9700 1 -- 172.21.15.6:0/351712582 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fc57804a8d0 con 0x7fc588102770 2024-02-09T21:50:34.214 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.634+0000 7fc5877fe700 1 --2- 172.21.15.6:0/351712582 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fc570036790 0x7fc570038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:34.214 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.634+0000 7fc5877fe700 1 --2- 172.21.15.6:0/351712582 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fc570036790 0x7fc570038c40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc57c009990 tx=0x7fc57c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:34.215 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.638+0000 7fc584ff9700 1 -- 172.21.15.6:0/351712582 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fc578023d60 con 0x7fc588102770 2024-02-09T21:50:34.215 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:32.887+0000 7fc58e922700 1 -- 172.21.15.6:0/351712582 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fc588077ff0 con 0x7fc588102770 2024-02-09T21:50:34.215 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:33.591+0000 7fc584ff9700 1 -- 172.21.15.6:0/351712582 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fc578024710 con 0x7fc588102770 2024-02-09T21:50:34.215 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:33.592+0000 7fc584ff9700 1 -- 172.21.15.6:0/351712582 <== mon.0 v2:172.21.15.6:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fc578018560 con 0x7fc588102770 2024-02-09T21:50:34.215 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:33.595+0000 7fc58e922700 1 -- 172.21.15.6:0/351712582 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fc570036790 msgr2=0x7fc570038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:34.215 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:33.595+0000 7fc58e922700 1 --2- 172.21.15.6:0/351712582 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fc570036790 0x7fc570038c40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc57c009990 tx=0x7fc57c006e40 comp rx=0 tx=0).stop 2024-02-09T21:50:34.215 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:33.595+0000 7fc58e922700 1 -- 172.21.15.6:0/351712582 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc588102770 msgr2=0x7fc588078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:34.215 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:33.595+0000 7fc58e922700 1 --2- 172.21.15.6:0/351712582 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc588102770 0x7fc588078d10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc5780090b0 tx=0x7fc578009250 comp rx=0 tx=0).stop 2024-02-09T21:50:34.215 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:33.595+0000 7fc58e922700 1 -- 172.21.15.6:0/351712582 shutdown_connections 2024-02-09T21:50:34.215 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:33.595+0000 7fc58e922700 1 --2- 172.21.15.6:0/351712582 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fc570036790 0x7fc570038c40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:34.215 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:33.595+0000 7fc58e922700 1 --2- 172.21.15.6:0/351712582 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc588102770 0x7fc588078d10 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:34.215 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:33.595+0000 7fc58e922700 1 -- 172.21.15.6:0/351712582 >> 172.21.15.6:0/351712582 conn(0x7fc5880fded0 msgr2=0x7fc5880ff5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:34.216 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:33.596+0000 7fc58e922700 1 -- 172.21.15.6:0/351712582 shutdown_connections 2024-02-09T21:50:34.216 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:33.596+0000 7fc58e922700 1 -- 172.21.15.6:0/351712582 wait complete. 2024-02-09T21:50:34.859 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:34 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/351712582' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-09T21:50:34.859 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:34 smithi006 ceph-mon[29258]: mgrmap e9: smithi006.glirxt(active, since 25s) 2024-02-09T21:50:35.486 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-02-09T21:50:35.486 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-02-09T21:50:35.486 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-09T21:50:35.486 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "active_name": "smithi006.glirxt", 2024-02-09T21:50:35.486 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-09T21:50:35.486 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-02-09T21:50:35.486 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.650+0000 7fed8609a700 1 Processor -- start 2024-02-09T21:50:35.486 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.650+0000 7fed8609a700 1 -- start start 2024-02-09T21:50:35.487 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.650+0000 7fed8609a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed80078d00 0x7fed80077210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:35.487 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.650+0000 7fed8609a700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed80079110 con 0x7fed80078d00 2024-02-09T21:50:35.487 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.650+0000 7fed7f7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed80078d00 0x7fed80077210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:35.487 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.650+0000 7fed7f7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed80078d00 0x7fed80077210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60072/0 (socket says 172.21.15.6:60072) 2024-02-09T21:50:35.487 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.650+0000 7fed7f7fe700 1 -- 172.21.15.6:0/4208657522 learned_addr learned my addr 172.21.15.6:0/4208657522 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:35.487 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.651+0000 7fed7f7fe700 1 -- 172.21.15.6:0/4208657522 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed80077750 con 0x7fed80078d00 2024-02-09T21:50:35.487 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.651+0000 7fed7f7fe700 1 --2- 172.21.15.6:0/4208657522 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed80078d00 0x7fed80077210 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fed700098e0 tx=0x7fed70009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c1e07339e5f2688 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:35.487 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.651+0000 7fed7e7fc700 1 -- 172.21.15.6:0/4208657522 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed7000b860 con 0x7fed80078d00 2024-02-09T21:50:35.487 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.651+0000 7fed7e7fc700 1 -- 172.21.15.6:0/4208657522 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fed7000be90 con 0x7fed80078d00 2024-02-09T21:50:35.487 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.651+0000 7fed7e7fc700 1 -- 172.21.15.6:0/4208657522 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed700103c0 con 0x7fed80078d00 2024-02-09T21:50:35.487 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.651+0000 7fed8609a700 1 -- 172.21.15.6:0/4208657522 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed80078d00 msgr2=0x7fed80077210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:35.487 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.651+0000 7fed8609a700 1 --2- 172.21.15.6:0/4208657522 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed80078d00 0x7fed80077210 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fed700098e0 tx=0x7fed70009250 comp rx=0 tx=0).stop 2024-02-09T21:50:35.487 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.651+0000 7fed8609a700 1 -- 172.21.15.6:0/4208657522 shutdown_connections 2024-02-09T21:50:35.487 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.651+0000 7fed8609a700 1 --2- 172.21.15.6:0/4208657522 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed80078d00 0x7fed80077210 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:35.488 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.652+0000 7fed8609a700 1 -- 172.21.15.6:0/4208657522 >> 172.21.15.6:0/4208657522 conn(0x7fed800fded0 msgr2=0x7fed80100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:35.488 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.652+0000 7fed8609a700 1 -- 172.21.15.6:0/4208657522 shutdown_connections 2024-02-09T21:50:35.488 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.652+0000 7fed8609a700 1 -- 172.21.15.6:0/4208657522 wait complete. 2024-02-09T21:50:35.488 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.652+0000 7fed8609a700 1 Processor -- start 2024-02-09T21:50:35.488 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.652+0000 7fed8609a700 1 -- start start 2024-02-09T21:50:35.488 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.652+0000 7fed8609a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed80078d00 0x7fed80113780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:35.488 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.652+0000 7fed8609a700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed80113cc0 con 0x7fed80078d00 2024-02-09T21:50:35.488 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.653+0000 7fed7f7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed80078d00 0x7fed80113780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:35.488 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.653+0000 7fed7f7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed80078d00 0x7fed80113780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60084/0 (socket says 172.21.15.6:60084) 2024-02-09T21:50:35.488 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.653+0000 7fed7f7fe700 1 -- 172.21.15.6:0/1353178602 learned_addr learned my addr 172.21.15.6:0/1353178602 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:35.488 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.653+0000 7fed7f7fe700 1 -- 172.21.15.6:0/1353178602 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed700094f0 con 0x7fed80078d00 2024-02-09T21:50:35.488 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.653+0000 7fed7f7fe700 1 --2- 172.21.15.6:0/1353178602 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed80078d00 0x7fed80113780 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fed70003080 tx=0x7fed70009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:35.488 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.653+0000 7fed67fff700 1 -- 172.21.15.6:0/1353178602 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed70011920 con 0x7fed80078d00 2024-02-09T21:50:35.489 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.653+0000 7fed8609a700 1 -- 172.21.15.6:0/1353178602 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed80113ec0 con 0x7fed80078d00 2024-02-09T21:50:35.489 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.653+0000 7fed67fff700 1 -- 172.21.15.6:0/1353178602 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fed70010d60 con 0x7fed80078d00 2024-02-09T21:50:35.489 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.653+0000 7fed67fff700 1 -- 172.21.15.6:0/1353178602 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed700243f0 con 0x7fed80078d00 2024-02-09T21:50:35.489 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.653+0000 7fed8609a700 1 -- 172.21.15.6:0/1353178602 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed80114b60 con 0x7fed80078d00 2024-02-09T21:50:35.489 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.654+0000 7fed67fff700 1 -- 172.21.15.6:0/1353178602 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fed70010890 con 0x7fed80078d00 2024-02-09T21:50:35.489 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.654+0000 7fed8609a700 1 -- 172.21.15.6:0/1353178602 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed8005f5c0 con 0x7fed80078d00 2024-02-09T21:50:35.489 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.655+0000 7fed67fff700 1 --2- 172.21.15.6:0/1353178602 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fed680367e0 0x7fed68038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:35.489 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.655+0000 7fed67fff700 1 -- 172.21.15.6:0/1353178602 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fed7004a3a0 con 0x7fed80078d00 2024-02-09T21:50:35.489 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.655+0000 7fed7effd700 1 -- 172.21.15.6:0/1353178602 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fed680367e0 msgr2=0x7fed68038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/1422219104 2024-02-09T21:50:35.489 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.655+0000 7fed7effd700 1 --2- 172.21.15.6:0/1353178602 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fed680367e0 0x7fed68038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-09T21:50:35.489 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.659+0000 7fed67fff700 1 -- 172.21.15.6:0/1353178602 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fed7002b940 con 0x7fed80078d00 2024-02-09T21:50:35.489 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.856+0000 7fed7effd700 1 -- 172.21.15.6:0/1353178602 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fed680367e0 msgr2=0x7fed68038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/1422219104 2024-02-09T21:50:35.489 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.856+0000 7fed7effd700 1 --2- 172.21.15.6:0/1353178602 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fed680367e0 0x7fed68038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-09T21:50:35.490 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.906+0000 7fed8609a700 1 -- 172.21.15.6:0/1353178602 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fed80114ea0 con 0x7fed80078d00 2024-02-09T21:50:35.490 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.907+0000 7fed67fff700 1 -- 172.21.15.6:0/1353178602 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7fed7002b760 con 0x7fed80078d00 2024-02-09T21:50:35.490 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.909+0000 7fed8609a700 1 -- 172.21.15.6:0/1353178602 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fed680367e0 msgr2=0x7fed68038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-09T21:50:35.490 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.909+0000 7fed8609a700 1 --2- 172.21.15.6:0/1353178602 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fed680367e0 0x7fed68038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:35.490 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.909+0000 7fed8609a700 1 -- 172.21.15.6:0/1353178602 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed80078d00 msgr2=0x7fed80113780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:35.490 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.909+0000 7fed8609a700 1 --2- 172.21.15.6:0/1353178602 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed80078d00 0x7fed80113780 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fed70003080 tx=0x7fed70009250 comp rx=0 tx=0).stop 2024-02-09T21:50:35.490 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.909+0000 7fed8609a700 1 -- 172.21.15.6:0/1353178602 shutdown_connections 2024-02-09T21:50:35.490 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.909+0000 7fed8609a700 1 --2- 172.21.15.6:0/1353178602 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7fed680367e0 0x7fed68038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:35.490 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.909+0000 7fed8609a700 1 --2- 172.21.15.6:0/1353178602 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed80078d00 0x7fed80113780 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:35.490 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.909+0000 7fed8609a700 1 -- 172.21.15.6:0/1353178602 >> 172.21.15.6:0/1353178602 conn(0x7fed800fded0 msgr2=0x7fed800ff5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:35.490 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.909+0000 7fed8609a700 1 -- 172.21.15.6:0/1353178602 shutdown_connections 2024-02-09T21:50:35.491 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:34.909+0000 7fed8609a700 1 -- 172.21.15.6:0/1353178602 wait complete. 2024-02-09T21:50:35.491 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for the mgr to restart... 2024-02-09T21:50:35.491 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for mgr epoch 9... 2024-02-09T21:50:35.755 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:35 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/1353178602' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-09T21:50:40.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:40 smithi006 ceph-mon[29258]: Active manager daemon smithi006.glirxt restarted 2024-02-09T21:50:40.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:40 smithi006 ceph-mon[29258]: Activating manager daemon smithi006.glirxt 2024-02-09T21:50:40.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:40 smithi006 ceph-mon[29258]: osdmap e3: 0 total, 0 up, 0 in 2024-02-09T21:50:40.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:40 smithi006 ceph-mon[29258]: mgrmap e10: smithi006.glirxt(active, starting, since 0.00317997s) 2024-02-09T21:50:40.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:40 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata", "id": "smithi006"}]: dispatch 2024-02-09T21:50:40.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:40 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mgr metadata", "who": "smithi006.glirxt", "id": "smithi006.glirxt"}]: dispatch 2024-02-09T21:50:40.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:40 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-09T21:50:40.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:40 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-09T21:50:40.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:40 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-09T21:50:40.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:40 smithi006 ceph-mon[29258]: Manager daemon smithi006.glirxt is now available 2024-02-09T21:50:40.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:40 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:50:40.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:40 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:50:41.560 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:41 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi006.glirxt/mirror_snapshot_schedule"}]: dispatch 2024-02-09T21:50:41.561 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:41 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi006.glirxt/trash_purge_schedule"}]: dispatch 2024-02-09T21:50:41.561 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:41 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:41.561 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:41 smithi006 ceph-mon[29258]: mgrmap e11: smithi006.glirxt(active, since 1.00803s) 2024-02-09T21:50:41.846 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-02-09T21:50:41.846 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-02-09T21:50:41.846 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-09T21:50:41.846 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-02-09T21:50:41.846 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.908+0000 7f86354a0700 1 Processor -- start 2024-02-09T21:50:41.847 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.908+0000 7f86354a0700 1 -- start start 2024-02-09T21:50:41.847 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.908+0000 7f86354a0700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86300714a0 0x7f8630105360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:41.847 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.908+0000 7f86354a0700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86301058a0 con 0x7f86300714a0 2024-02-09T21:50:41.847 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.908+0000 7f862effd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86300714a0 0x7f8630105360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:41.847 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.908+0000 7f862effd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86300714a0 0x7f8630105360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60094/0 (socket says 172.21.15.6:60094) 2024-02-09T21:50:41.847 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.908+0000 7f862effd700 1 -- 172.21.15.6:0/1513672977 learned_addr learned my addr 172.21.15.6:0/1513672977 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:41.847 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.908+0000 7f862effd700 1 -- 172.21.15.6:0/1513672977 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86301059e0 con 0x7f86300714a0 2024-02-09T21:50:41.847 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.908+0000 7f862effd700 1 --2- 172.21.15.6:0/1513672977 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86300714a0 0x7f8630105360 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f86200098e0 tx=0x7f8620009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6028643ef6faa071 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:41.847 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.909+0000 7f862dffb700 1 -- 172.21.15.6:0/1513672977 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f862000b860 con 0x7f86300714a0 2024-02-09T21:50:41.847 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.909+0000 7f862dffb700 1 -- 172.21.15.6:0/1513672977 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f862000be90 con 0x7f86300714a0 2024-02-09T21:50:41.847 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.909+0000 7f862dffb700 1 -- 172.21.15.6:0/1513672977 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86200103c0 con 0x7f86300714a0 2024-02-09T21:50:41.847 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.909+0000 7f86354a0700 1 -- 172.21.15.6:0/1513672977 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86300714a0 msgr2=0x7f8630105360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:41.847 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.909+0000 7f86354a0700 1 --2- 172.21.15.6:0/1513672977 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86300714a0 0x7f8630105360 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f86200098e0 tx=0x7f8620009250 comp rx=0 tx=0).stop 2024-02-09T21:50:41.847 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.909+0000 7f86354a0700 1 -- 172.21.15.6:0/1513672977 shutdown_connections 2024-02-09T21:50:41.847 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.909+0000 7f86354a0700 1 --2- 172.21.15.6:0/1513672977 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86300714a0 0x7f8630105360 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:41.848 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.909+0000 7f86354a0700 1 -- 172.21.15.6:0/1513672977 >> 172.21.15.6:0/1513672977 conn(0x7f86300fcdd0 msgr2=0x7f86300ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:41.848 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.910+0000 7f86354a0700 1 -- 172.21.15.6:0/1513672977 shutdown_connections 2024-02-09T21:50:41.848 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.910+0000 7f86354a0700 1 -- 172.21.15.6:0/1513672977 wait complete. 2024-02-09T21:50:41.848 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.910+0000 7f86354a0700 1 Processor -- start 2024-02-09T21:50:41.848 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.910+0000 7f86354a0700 1 -- start start 2024-02-09T21:50:41.848 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.910+0000 7f86354a0700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86300714a0 0x7f86301115c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:41.848 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.910+0000 7f86354a0700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8630111b00 con 0x7f86300714a0 2024-02-09T21:50:41.848 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.911+0000 7f862effd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86300714a0 0x7f86301115c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:41.848 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.911+0000 7f862effd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86300714a0 0x7f86301115c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60108/0 (socket says 172.21.15.6:60108) 2024-02-09T21:50:41.848 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.911+0000 7f862effd700 1 -- 172.21.15.6:0/370100724 learned_addr learned my addr 172.21.15.6:0/370100724 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:41.848 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.911+0000 7f862effd700 1 -- 172.21.15.6:0/370100724 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86200094f0 con 0x7f86300714a0 2024-02-09T21:50:41.848 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.911+0000 7f862effd700 1 --2- 172.21.15.6:0/370100724 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86300714a0 0x7f86301115c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8620006d40 tx=0x7f8620009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:41.848 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.911+0000 7f86177fe700 1 -- 172.21.15.6:0/370100724 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8620010840 con 0x7f86300714a0 2024-02-09T21:50:41.848 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.911+0000 7f86177fe700 1 -- 172.21.15.6:0/370100724 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8620010e70 con 0x7f86300714a0 2024-02-09T21:50:41.849 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.911+0000 7f86354a0700 1 -- 172.21.15.6:0/370100724 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8630111d00 con 0x7f86300714a0 2024-02-09T21:50:41.849 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.911+0000 7f86177fe700 1 -- 172.21.15.6:0/370100724 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8620024ea0 con 0x7f86300714a0 2024-02-09T21:50:41.849 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.911+0000 7f86354a0700 1 -- 172.21.15.6:0/370100724 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86301129a0 con 0x7f86300714a0 2024-02-09T21:50:41.849 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.912+0000 7f86177fe700 1 -- 172.21.15.6:0/370100724 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f862001a4c0 con 0x7f86300714a0 2024-02-09T21:50:41.849 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.912+0000 7f86177fe700 1 --2- 172.21.15.6:0/370100724 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f86180367e0 0x7f8618038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:41.849 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.912+0000 7f86177fe700 1 -- 172.21.15.6:0/370100724 --> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f86180393a0 con 0x7f86180367e0 2024-02-09T21:50:41.849 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.912+0000 7f86177fe700 1 -- 172.21.15.6:0/370100724 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f862004a730 con 0x7f86300714a0 2024-02-09T21:50:41.849 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.912+0000 7f862e7fc700 1 -- 172.21.15.6:0/370100724 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f86180367e0 msgr2=0x7f8618038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/1422219104 2024-02-09T21:50:41.849 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:35.913+0000 7f862e7fc700 1 --2- 172.21.15.6:0/370100724 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f86180367e0 0x7f8618038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-09T21:50:41.849 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:36.114+0000 7f862e7fc700 1 -- 172.21.15.6:0/370100724 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f86180367e0 msgr2=0x7f8618038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/1422219104 2024-02-09T21:50:41.849 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:36.114+0000 7f862e7fc700 1 --2- 172.21.15.6:0/370100724 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f86180367e0 0x7f8618038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-09T21:50:41.849 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:36.515+0000 7f862e7fc700 1 -- 172.21.15.6:0/370100724 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f86180367e0 msgr2=0x7f8618038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/1422219104 2024-02-09T21:50:41.849 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:36.515+0000 7f862e7fc700 1 --2- 172.21.15.6:0/370100724 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f86180367e0 0x7f8618038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-09T21:50:41.849 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:37.316+0000 7f862e7fc700 1 -- 172.21.15.6:0/370100724 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f86180367e0 msgr2=0x7f8618038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/1422219104 2024-02-09T21:50:41.850 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:37.316+0000 7f862e7fc700 1 --2- 172.21.15.6:0/370100724 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f86180367e0 0x7f8618038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-09T21:50:41.850 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:38.917+0000 7f862e7fc700 1 -- 172.21.15.6:0/370100724 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f86180367e0 msgr2=0x7f8618038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/1422219104 2024-02-09T21:50:41.850 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:38.917+0000 7f862e7fc700 1 --2- 172.21.15.6:0/370100724 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f86180367e0 0x7f8618038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-02-09T21:50:41.850 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:40.246+0000 7f86177fe700 1 -- 172.21.15.6:0/370100724 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mgrmap(e 10) v1 ==== 43148+0+0 (secure 0 0 0) 0x7f8620024780 con 0x7f86300714a0 2024-02-09T21:50:41.850 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:40.247+0000 7f86177fe700 1 -- 172.21.15.6:0/370100724 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f86180367e0 msgr2=0x7f8618038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-09T21:50:41.850 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:40.247+0000 7f86177fe700 1 --2- 172.21.15.6:0/370100724 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f86180367e0 0x7f8618038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:41.850 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:41.251+0000 7f86177fe700 1 -- 172.21.15.6:0/370100724 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f8620024430 con 0x7f86300714a0 2024-02-09T21:50:41.850 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:41.251+0000 7f86177fe700 1 --2- 172.21.15.6:0/370100724 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f8618039fd0 0x7f861803c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:41.850 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:41.251+0000 7f86177fe700 1 -- 172.21.15.6:0/370100724 --> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f86180393a0 con 0x7f8618039fd0 2024-02-09T21:50:41.850 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:41.251+0000 7f862e7fc700 1 --2- 172.21.15.6:0/370100724 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f8618039fd0 0x7f861803c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:41.850 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:41.252+0000 7f862e7fc700 1 --2- 172.21.15.6:0/370100724 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f8618039fd0 0x7f861803c3b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f86240039d0 tx=0x7f8624009350 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:41.850 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:41.253+0000 7f86177fe700 1 -- 172.21.15.6:0/370100724 <== mgr.14162 v2:172.21.15.6:6800/3321564517 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f86180393a0 con 0x7f8618039fd0 2024-02-09T21:50:41.850 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:41.258+0000 7f86354a0700 1 -- 172.21.15.6:0/370100724 --> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f8630112190 con 0x7f8618039fd0 2024-02-09T21:50:41.850 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:41.259+0000 7f86177fe700 1 -- 172.21.15.6:0/370100724 <== mgr.14162 v2:172.21.15.6:6800/3321564517 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f8630112190 con 0x7f8618039fd0 2024-02-09T21:50:41.850 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:41.259+0000 7f86354a0700 1 -- 172.21.15.6:0/370100724 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f8618039fd0 msgr2=0x7f861803c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:41.851 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:41.259+0000 7f86354a0700 1 --2- 172.21.15.6:0/370100724 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f8618039fd0 0x7f861803c3b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f86240039d0 tx=0x7f8624009350 comp rx=0 tx=0).stop 2024-02-09T21:50:41.851 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:41.259+0000 7f86354a0700 1 -- 172.21.15.6:0/370100724 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86300714a0 msgr2=0x7f86301115c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:41.851 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:41.259+0000 7f86354a0700 1 --2- 172.21.15.6:0/370100724 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86300714a0 0x7f86301115c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8620006d40 tx=0x7f8620009250 comp rx=0 tx=0).stop 2024-02-09T21:50:41.851 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:41.259+0000 7f86354a0700 1 -- 172.21.15.6:0/370100724 shutdown_connections 2024-02-09T21:50:41.851 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:41.259+0000 7f86354a0700 1 --2- 172.21.15.6:0/370100724 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86300714a0 0x7f86301115c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:41.851 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:41.259+0000 7f86354a0700 1 --2- 172.21.15.6:0/370100724 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f8618039fd0 0x7f861803c3b0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:41.851 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:41.259+0000 7f86354a0700 1 --2- 172.21.15.6:0/370100724 >> [v2:172.21.15.6:6800/1422219104,v1:172.21.15.6:6801/1422219104] conn(0x7f86180367e0 0x7f8618038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:41.851 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:41.259+0000 7f86354a0700 1 -- 172.21.15.6:0/370100724 >> 172.21.15.6:0/370100724 conn(0x7f86300fcdd0 msgr2=0x7f86300fe540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:41.851 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:41.259+0000 7f86354a0700 1 -- 172.21.15.6:0/370100724 shutdown_connections 2024-02-09T21:50:41.851 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:41.259+0000 7f86354a0700 1 -- 172.21.15.6:0/370100724 wait complete. 2024-02-09T21:50:41.851 INFO:teuthology.orchestra.run.smithi006.stdout:mgr epoch 9 is available 2024-02-09T21:50:41.851 INFO:teuthology.orchestra.run.smithi006.stdout:Generating a dashboard self-signed certificate... 2024-02-09T21:50:42.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:42 smithi006 ceph-mon[29258]: [09/Feb/2024:21:50:40] ENGINE Bus STARTING 2024-02-09T21:50:42.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:42 smithi006 ceph-mon[29258]: [09/Feb/2024:21:50:41] ENGINE Serving on https://172.21.15.6:7150 2024-02-09T21:50:42.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:42 smithi006 ceph-mon[29258]: [09/Feb/2024:21:50:41] ENGINE Bus STARTED 2024-02-09T21:50:42.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:42 smithi006 ceph-mon[29258]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-09T21:50:42.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:42 smithi006 ceph-mon[29258]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-09T21:50:43.335 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:43 smithi006 ceph-mon[29258]: mgrmap e12: smithi006.glirxt(active, since 2s) 2024-02-09T21:50:43.335 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:43 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:43.336 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:43 smithi006 ceph-mon[29258]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:50:43.336 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:43 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:43.336 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:43 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:43.336 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:43 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:43.343 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-02-09T21:50:43.343 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.253+0000 7fab68bdc700 1 Processor -- start 2024-02-09T21:50:43.343 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.253+0000 7fab68bdc700 1 -- start start 2024-02-09T21:50:43.343 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.253+0000 7fab68bdc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab64102d70 0x7fab64103180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:43.343 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.253+0000 7fab68bdc700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab641036c0 con 0x7fab64102d70 2024-02-09T21:50:43.343 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.253+0000 7fab6259c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab64102d70 0x7fab64103180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:43.343 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.253+0000 7fab6259c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab64102d70 0x7fab64103180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50546/0 (socket says 172.21.15.6:50546) 2024-02-09T21:50:43.343 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.253+0000 7fab6259c700 1 -- 172.21.15.6:0/2918907141 learned_addr learned my addr 172.21.15.6:0/2918907141 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:43.344 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.253+0000 7fab6259c700 1 -- 172.21.15.6:0/2918907141 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab64103800 con 0x7fab64102d70 2024-02-09T21:50:43.344 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.254+0000 7fab6259c700 1 --2- 172.21.15.6:0/2918907141 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab64102d70 0x7fab64103180 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fab540098e0 tx=0x7fab54009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fecab7ab62992e5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:43.344 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.254+0000 7fab6159a700 1 -- 172.21.15.6:0/2918907141 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab5400b8f0 con 0x7fab64102d70 2024-02-09T21:50:43.344 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.254+0000 7fab6159a700 1 -- 172.21.15.6:0/2918907141 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fab54011420 con 0x7fab64102d70 2024-02-09T21:50:43.344 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.254+0000 7fab6159a700 1 -- 172.21.15.6:0/2918907141 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab54010540 con 0x7fab64102d70 2024-02-09T21:50:43.344 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.254+0000 7fab68bdc700 1 -- 172.21.15.6:0/2918907141 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab64102d70 msgr2=0x7fab64103180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:43.344 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.254+0000 7fab68bdc700 1 --2- 172.21.15.6:0/2918907141 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab64102d70 0x7fab64103180 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fab540098e0 tx=0x7fab54009250 comp rx=0 tx=0).stop 2024-02-09T21:50:43.344 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.254+0000 7fab68bdc700 1 -- 172.21.15.6:0/2918907141 shutdown_connections 2024-02-09T21:50:43.344 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.254+0000 7fab68bdc700 1 --2- 172.21.15.6:0/2918907141 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab64102d70 0x7fab64103180 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:43.344 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.254+0000 7fab68bdc700 1 -- 172.21.15.6:0/2918907141 >> 172.21.15.6:0/2918907141 conn(0x7fab640fc580 msgr2=0x7fab640fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:43.344 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.255+0000 7fab68bdc700 1 -- 172.21.15.6:0/2918907141 shutdown_connections 2024-02-09T21:50:43.344 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.255+0000 7fab68bdc700 1 -- 172.21.15.6:0/2918907141 wait complete. 2024-02-09T21:50:43.344 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.255+0000 7fab68bdc700 1 Processor -- start 2024-02-09T21:50:43.344 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.255+0000 7fab68bdc700 1 -- start start 2024-02-09T21:50:43.345 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.255+0000 7fab68bdc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab64102d70 0x7fab64113a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:43.345 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.255+0000 7fab68bdc700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab64113f90 con 0x7fab64102d70 2024-02-09T21:50:43.345 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.255+0000 7fab6259c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab64102d70 0x7fab64113a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:43.345 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.255+0000 7fab6259c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab64102d70 0x7fab64113a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50552/0 (socket says 172.21.15.6:50552) 2024-02-09T21:50:43.345 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.255+0000 7fab6259c700 1 -- 172.21.15.6:0/856005132 learned_addr learned my addr 172.21.15.6:0/856005132 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:43.345 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.256+0000 7fab6259c700 1 -- 172.21.15.6:0/856005132 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab540094f0 con 0x7fab64102d70 2024-02-09T21:50:43.345 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.256+0000 7fab6259c700 1 --2- 172.21.15.6:0/856005132 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab64102d70 0x7fab64113a50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fab54011a90 tx=0x7fab54009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:43.345 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.256+0000 7fab52ffd700 1 -- 172.21.15.6:0/856005132 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab540107b0 con 0x7fab64102d70 2024-02-09T21:50:43.345 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.256+0000 7fab52ffd700 1 -- 172.21.15.6:0/856005132 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fab54010de0 con 0x7fab64102d70 2024-02-09T21:50:43.345 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.256+0000 7fab68bdc700 1 -- 172.21.15.6:0/856005132 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab64114190 con 0x7fab64102d70 2024-02-09T21:50:43.345 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.256+0000 7fab52ffd700 1 -- 172.21.15.6:0/856005132 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab5401ae40 con 0x7fab64102d70 2024-02-09T21:50:43.345 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.256+0000 7fab68bdc700 1 -- 172.21.15.6:0/856005132 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab64114e30 con 0x7fab64102d70 2024-02-09T21:50:43.345 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.257+0000 7fab52ffd700 1 -- 172.21.15.6:0/856005132 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7fab54010910 con 0x7fab64102d70 2024-02-09T21:50:43.345 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.257+0000 7fab68bdc700 1 -- 172.21.15.6:0/856005132 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab6405f5c0 con 0x7fab64102d70 2024-02-09T21:50:43.345 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.257+0000 7fab52ffd700 1 --2- 172.21.15.6:0/856005132 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fab4c036790 0x7fab4c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:43.346 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.257+0000 7fab52ffd700 1 -- 172.21.15.6:0/856005132 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fab5404a430 con 0x7fab64102d70 2024-02-09T21:50:43.346 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.257+0000 7fab61d9b700 1 --2- 172.21.15.6:0/856005132 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fab4c036790 0x7fab4c038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:43.346 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.258+0000 7fab61d9b700 1 --2- 172.21.15.6:0/856005132 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fab4c036790 0x7fab4c038c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fab58009990 tx=0x7fab58006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:43.346 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.262+0000 7fab52ffd700 1 -- 172.21.15.6:0/856005132 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fab54018b40 con 0x7fab64102d70 2024-02-09T21:50:43.346 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.301+0000 7fab52ffd700 1 -- 172.21.15.6:0/856005132 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fab5402e960 con 0x7fab64102d70 2024-02-09T21:50:43.346 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.468+0000 7fab68bdc700 1 -- 172.21.15.6:0/856005132 --> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fab64108b20 con 0x7fab4c036790 2024-02-09T21:50:43.346 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.688+0000 7fab52ffd700 1 -- 172.21.15.6:0/856005132 <== mgr.14162 v2:172.21.15.6:6800/3321564517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fab64108b20 con 0x7fab4c036790 2024-02-09T21:50:43.346 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.691+0000 7fab68bdc700 1 -- 172.21.15.6:0/856005132 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fab4c036790 msgr2=0x7fab4c038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:43.346 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.691+0000 7fab68bdc700 1 --2- 172.21.15.6:0/856005132 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fab4c036790 0x7fab4c038c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fab58009990 tx=0x7fab58006e40 comp rx=0 tx=0).stop 2024-02-09T21:50:43.346 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.691+0000 7fab68bdc700 1 -- 172.21.15.6:0/856005132 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab64102d70 msgr2=0x7fab64113a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:43.346 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.691+0000 7fab68bdc700 1 --2- 172.21.15.6:0/856005132 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab64102d70 0x7fab64113a50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fab54011a90 tx=0x7fab54009250 comp rx=0 tx=0).stop 2024-02-09T21:50:43.346 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.691+0000 7fab68bdc700 1 -- 172.21.15.6:0/856005132 shutdown_connections 2024-02-09T21:50:43.346 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.691+0000 7fab68bdc700 1 --2- 172.21.15.6:0/856005132 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fab4c036790 0x7fab4c038c40 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:43.346 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.691+0000 7fab68bdc700 1 --2- 172.21.15.6:0/856005132 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab64102d70 0x7fab64113a50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:43.347 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.691+0000 7fab68bdc700 1 -- 172.21.15.6:0/856005132 >> 172.21.15.6:0/856005132 conn(0x7fab640fc580 msgr2=0x7fab640fd410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:43.347 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.691+0000 7fab68bdc700 1 -- 172.21.15.6:0/856005132 shutdown_connections 2024-02-09T21:50:43.347 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:42.691+0000 7fab68bdc700 1 -- 172.21.15.6:0/856005132 wait complete. 2024-02-09T21:50:43.347 INFO:teuthology.orchestra.run.smithi006.stdout:Creating initial admin user... 2024-02-09T21:50:44.854 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$kUTYu/H6lMJKHuCpoB8L6uHXT4AdMZ9/zRCX1Vf9O7H0eaZNCxvrK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1707515444, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-09T21:50:44.854 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.755+0000 7f03427ee700 1 Processor -- start 2024-02-09T21:50:44.854 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.755+0000 7f03427ee700 1 -- start start 2024-02-09T21:50:44.854 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.755+0000 7f03427ee700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f033c071480 0x7f033c105380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:44.854 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.755+0000 7f03427ee700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033c1058c0 con 0x7f033c071480 2024-02-09T21:50:44.854 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.755+0000 7f033bfff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f033c071480 0x7f033c105380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:44.855 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.755+0000 7f033bfff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f033c071480 0x7f033c105380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50564/0 (socket says 172.21.15.6:50564) 2024-02-09T21:50:44.855 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.755+0000 7f033bfff700 1 -- 172.21.15.6:0/1758317239 learned_addr learned my addr 172.21.15.6:0/1758317239 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:44.855 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.755+0000 7f033bfff700 1 -- 172.21.15.6:0/1758317239 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f033c105a00 con 0x7f033c071480 2024-02-09T21:50:44.855 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.755+0000 7f033bfff700 1 --2- 172.21.15.6:0/1758317239 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f033c071480 0x7f033c105380 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f032c0098e0 tx=0x7f032c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3256154099c2dd13 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:44.855 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.756+0000 7f033affd700 1 -- 172.21.15.6:0/1758317239 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f032c00b860 con 0x7f033c071480 2024-02-09T21:50:44.855 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.756+0000 7f033affd700 1 -- 172.21.15.6:0/1758317239 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f032c00be90 con 0x7f033c071480 2024-02-09T21:50:44.855 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.756+0000 7f033affd700 1 -- 172.21.15.6:0/1758317239 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f032c0103c0 con 0x7f033c071480 2024-02-09T21:50:44.855 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.757+0000 7f03427ee700 1 -- 172.21.15.6:0/1758317239 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f033c071480 msgr2=0x7f033c105380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:44.855 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.757+0000 7f03427ee700 1 --2- 172.21.15.6:0/1758317239 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f033c071480 0x7f033c105380 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f032c0098e0 tx=0x7f032c009250 comp rx=0 tx=0).stop 2024-02-09T21:50:44.855 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.757+0000 7f03427ee700 1 -- 172.21.15.6:0/1758317239 shutdown_connections 2024-02-09T21:50:44.855 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.757+0000 7f03427ee700 1 --2- 172.21.15.6:0/1758317239 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f033c071480 0x7f033c105380 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:44.855 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.757+0000 7f03427ee700 1 -- 172.21.15.6:0/1758317239 >> 172.21.15.6:0/1758317239 conn(0x7f033c0fcdd0 msgr2=0x7f033c0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:44.855 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.757+0000 7f03427ee700 1 -- 172.21.15.6:0/1758317239 shutdown_connections 2024-02-09T21:50:44.855 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.757+0000 7f03427ee700 1 -- 172.21.15.6:0/1758317239 wait complete. 2024-02-09T21:50:44.856 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.757+0000 7f03427ee700 1 Processor -- start 2024-02-09T21:50:44.856 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.757+0000 7f03427ee700 1 -- start start 2024-02-09T21:50:44.856 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.758+0000 7f03427ee700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f033c071480 0x7f033c1115e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:44.856 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.758+0000 7f03427ee700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033c111b20 con 0x7f033c071480 2024-02-09T21:50:44.856 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.758+0000 7f033bfff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f033c071480 0x7f033c1115e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:44.856 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.758+0000 7f033bfff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f033c071480 0x7f033c1115e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50572/0 (socket says 172.21.15.6:50572) 2024-02-09T21:50:44.856 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.758+0000 7f033bfff700 1 -- 172.21.15.6:0/823464752 learned_addr learned my addr 172.21.15.6:0/823464752 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:44.856 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.758+0000 7f033bfff700 1 -- 172.21.15.6:0/823464752 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f032c0094f0 con 0x7f033c071480 2024-02-09T21:50:44.856 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.758+0000 7f033bfff700 1 --2- 172.21.15.6:0/823464752 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f033c071480 0x7f033c1115e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f032c006d40 tx=0x7f032c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:44.856 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.758+0000 7f0338ff9700 1 -- 172.21.15.6:0/823464752 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f032c010840 con 0x7f033c071480 2024-02-09T21:50:44.856 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.758+0000 7f0338ff9700 1 -- 172.21.15.6:0/823464752 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f032c010e70 con 0x7f033c071480 2024-02-09T21:50:44.856 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.758+0000 7f03427ee700 1 -- 172.21.15.6:0/823464752 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f033c111d20 con 0x7f033c071480 2024-02-09T21:50:44.856 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.758+0000 7f0338ff9700 1 -- 172.21.15.6:0/823464752 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f032c024ea0 con 0x7f033c071480 2024-02-09T21:50:44.857 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.759+0000 7f03427ee700 1 -- 172.21.15.6:0/823464752 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f033c1129c0 con 0x7f033c071480 2024-02-09T21:50:44.857 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.760+0000 7f0338ff9700 1 -- 172.21.15.6:0/823464752 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f032c01a510 con 0x7f033c071480 2024-02-09T21:50:44.857 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.760+0000 7f0338ff9700 1 --2- 172.21.15.6:0/823464752 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f0324036790 0x7f0324038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:44.857 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.760+0000 7f03427ee700 1 -- 172.21.15.6:0/823464752 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f033c05f5c0 con 0x7f033c071480 2024-02-09T21:50:44.857 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.760+0000 7f033b7fe700 1 --2- 172.21.15.6:0/823464752 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f0324036790 0x7f0324038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:44.857 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.760+0000 7f0338ff9700 1 -- 172.21.15.6:0/823464752 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f032c04a810 con 0x7f033c071480 2024-02-09T21:50:44.857 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.760+0000 7f033b7fe700 1 --2- 172.21.15.6:0/823464752 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f0324036790 0x7f0324038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0330009990 tx=0x7f0330006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:44.857 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.765+0000 7f0338ff9700 1 -- 172.21.15.6:0/823464752 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f032c018710 con 0x7f033c071480 2024-02-09T21:50:44.857 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:43.974+0000 7f03427ee700 1 -- 172.21.15.6:0/823464752 --> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f033c05f3e0 con 0x7f0324036790 2024-02-09T21:50:44.857 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:44.221+0000 7f0338ff9700 1 -- 172.21.15.6:0/823464752 <== mgr.14162 v2:172.21.15.6:6800/3321564517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f033c05f3e0 con 0x7f0324036790 2024-02-09T21:50:44.857 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:44.223+0000 7f03427ee700 1 -- 172.21.15.6:0/823464752 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f0324036790 msgr2=0x7f0324038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:44.857 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:44.223+0000 7f03427ee700 1 --2- 172.21.15.6:0/823464752 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f0324036790 0x7f0324038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0330009990 tx=0x7f0330006e40 comp rx=0 tx=0).stop 2024-02-09T21:50:44.858 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:44.223+0000 7f03427ee700 1 -- 172.21.15.6:0/823464752 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f033c071480 msgr2=0x7f033c1115e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:44.858 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:44.223+0000 7f03427ee700 1 --2- 172.21.15.6:0/823464752 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f033c071480 0x7f033c1115e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f032c006d40 tx=0x7f032c009250 comp rx=0 tx=0).stop 2024-02-09T21:50:44.858 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:44.223+0000 7f03427ee700 1 -- 172.21.15.6:0/823464752 shutdown_connections 2024-02-09T21:50:44.858 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:44.223+0000 7f03427ee700 1 --2- 172.21.15.6:0/823464752 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f0324036790 0x7f0324038c40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:44.858 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:44.223+0000 7f03427ee700 1 --2- 172.21.15.6:0/823464752 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f033c071480 0x7f033c1115e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:44.858 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:44.223+0000 7f03427ee700 1 -- 172.21.15.6:0/823464752 >> 172.21.15.6:0/823464752 conn(0x7f033c0fcdd0 msgr2=0x7f033c0fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:44.858 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:44.223+0000 7f03427ee700 1 -- 172.21.15.6:0/823464752 shutdown_connections 2024-02-09T21:50:44.858 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:44.224+0000 7f03427ee700 1 -- 172.21.15.6:0/823464752 wait complete. 2024-02-09T21:50:44.858 INFO:teuthology.orchestra.run.smithi006.stdout:Fetching dashboard port number... 2024-02-09T21:50:45.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:45 smithi006 ceph-mon[29258]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:50:45.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:45 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:46.338 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:46 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/1951139128' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-09T21:50:46.346 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 8443 2024-02-09T21:50:46.346 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.285+0000 7f9456663700 1 Processor -- start 2024-02-09T21:50:46.346 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.286+0000 7f9456663700 1 -- start start 2024-02-09T21:50:46.346 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.286+0000 7f9456663700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9450102770 0x7f9450102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:46.346 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.286+0000 7f9456663700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94501030c0 con 0x7f9450102770 2024-02-09T21:50:46.346 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.286+0000 7f944ffff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9450102770 0x7f9450102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:46.346 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.286+0000 7f944ffff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9450102770 0x7f9450102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50576/0 (socket says 172.21.15.6:50576) 2024-02-09T21:50:46.346 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.286+0000 7f944ffff700 1 -- 172.21.15.6:0/3423851793 learned_addr learned my addr 172.21.15.6:0/3423851793 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:46.346 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.286+0000 7f944ffff700 1 -- 172.21.15.6:0/3423851793 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9450103200 con 0x7f9450102770 2024-02-09T21:50:46.346 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.286+0000 7f944ffff700 1 --2- 172.21.15.6:0/3423851793 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9450102770 0x7f9450102b80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f94380098e0 tx=0x7f9438009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=749289f32ef53ab5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:46.346 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.286+0000 7f944effd700 1 -- 172.21.15.6:0/3423851793 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f943800b860 con 0x7f9450102770 2024-02-09T21:50:46.346 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.287+0000 7f944effd700 1 -- 172.21.15.6:0/3423851793 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f943800be90 con 0x7f9450102770 2024-02-09T21:50:46.347 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.287+0000 7f944effd700 1 -- 172.21.15.6:0/3423851793 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94380103c0 con 0x7f9450102770 2024-02-09T21:50:46.347 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.287+0000 7f9456663700 1 -- 172.21.15.6:0/3423851793 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9450102770 msgr2=0x7f9450102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:46.347 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.287+0000 7f9456663700 1 --2- 172.21.15.6:0/3423851793 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9450102770 0x7f9450102b80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f94380098e0 tx=0x7f9438009250 comp rx=0 tx=0).stop 2024-02-09T21:50:46.347 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.287+0000 7f9456663700 1 -- 172.21.15.6:0/3423851793 shutdown_connections 2024-02-09T21:50:46.347 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.287+0000 7f9456663700 1 --2- 172.21.15.6:0/3423851793 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9450102770 0x7f9450102b80 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:46.347 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.287+0000 7f9456663700 1 -- 172.21.15.6:0/3423851793 >> 172.21.15.6:0/3423851793 conn(0x7f94500fded0 msgr2=0x7f9450100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:46.347 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.287+0000 7f9456663700 1 -- 172.21.15.6:0/3423851793 shutdown_connections 2024-02-09T21:50:46.347 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.287+0000 7f9456663700 1 -- 172.21.15.6:0/3423851793 wait complete. 2024-02-09T21:50:46.347 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.288+0000 7f9456663700 1 Processor -- start 2024-02-09T21:50:46.347 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.288+0000 7f9456663700 1 -- start start 2024-02-09T21:50:46.347 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.288+0000 7f9456663700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9450102770 0x7f94501115c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:46.347 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.288+0000 7f9456663700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9450111b00 con 0x7f9450102770 2024-02-09T21:50:46.347 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.288+0000 7f944ffff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9450102770 0x7f94501115c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:46.347 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.288+0000 7f944ffff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9450102770 0x7f94501115c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50578/0 (socket says 172.21.15.6:50578) 2024-02-09T21:50:46.347 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.288+0000 7f944ffff700 1 -- 172.21.15.6:0/1951139128 learned_addr learned my addr 172.21.15.6:0/1951139128 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:46.348 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.289+0000 7f944ffff700 1 -- 172.21.15.6:0/1951139128 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94380094f0 con 0x7f9450102770 2024-02-09T21:50:46.348 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.289+0000 7f944ffff700 1 --2- 172.21.15.6:0/1951139128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9450102770 0x7f94501115c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f94380090b0 tx=0x7f9438009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:46.348 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.289+0000 7f944cff9700 1 -- 172.21.15.6:0/1951139128 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9438010840 con 0x7f9450102770 2024-02-09T21:50:46.348 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.289+0000 7f944cff9700 1 -- 172.21.15.6:0/1951139128 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9438010e70 con 0x7f9450102770 2024-02-09T21:50:46.348 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.289+0000 7f944cff9700 1 -- 172.21.15.6:0/1951139128 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9438024e40 con 0x7f9450102770 2024-02-09T21:50:46.348 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.289+0000 7f9456663700 1 -- 172.21.15.6:0/1951139128 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9450111d00 con 0x7f9450102770 2024-02-09T21:50:46.348 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.289+0000 7f9456663700 1 -- 172.21.15.6:0/1951139128 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94501129a0 con 0x7f9450102770 2024-02-09T21:50:46.348 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.290+0000 7f9456663700 1 -- 172.21.15.6:0/1951139128 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f945005f5c0 con 0x7f9450102770 2024-02-09T21:50:46.348 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.290+0000 7f944cff9700 1 -- 172.21.15.6:0/1951139128 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f94380109a0 con 0x7f9450102770 2024-02-09T21:50:46.348 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.290+0000 7f944cff9700 1 --2- 172.21.15.6:0/1951139128 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f943c0367e0 0x7f943c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:46.348 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.290+0000 7f944cff9700 1 -- 172.21.15.6:0/1951139128 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f943804a850 con 0x7f9450102770 2024-02-09T21:50:46.348 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.291+0000 7f944f7fe700 1 --2- 172.21.15.6:0/1951139128 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f943c0367e0 0x7f943c038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:46.348 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.291+0000 7f944f7fe700 1 --2- 172.21.15.6:0/1951139128 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f943c0367e0 0x7f943c038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9440009990 tx=0x7f9440006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:46.348 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.299+0000 7f944cff9700 1 -- 172.21.15.6:0/1951139128 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f943802e440 con 0x7f9450102770 2024-02-09T21:50:46.349 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.516+0000 7f9456663700 1 -- 172.21.15.6:0/1951139128 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f945004bed0 con 0x7f9450102770 2024-02-09T21:50:46.349 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.517+0000 7f944cff9700 1 -- 172.21.15.6:0/1951139128 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f943802c340 con 0x7f9450102770 2024-02-09T21:50:46.349 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.518+0000 7f9456663700 1 -- 172.21.15.6:0/1951139128 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f943c0367e0 msgr2=0x7f943c038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:46.349 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.518+0000 7f9456663700 1 --2- 172.21.15.6:0/1951139128 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f943c0367e0 0x7f943c038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9440009990 tx=0x7f9440006e40 comp rx=0 tx=0).stop 2024-02-09T21:50:46.349 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.518+0000 7f9456663700 1 -- 172.21.15.6:0/1951139128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9450102770 msgr2=0x7f94501115c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:46.349 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.518+0000 7f9456663700 1 --2- 172.21.15.6:0/1951139128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9450102770 0x7f94501115c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f94380090b0 tx=0x7f9438009250 comp rx=0 tx=0).stop 2024-02-09T21:50:46.349 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.518+0000 7f9456663700 1 -- 172.21.15.6:0/1951139128 shutdown_connections 2024-02-09T21:50:46.349 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.519+0000 7f9456663700 1 --2- 172.21.15.6:0/1951139128 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f943c0367e0 0x7f943c038c90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:46.349 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.519+0000 7f9456663700 1 --2- 172.21.15.6:0/1951139128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9450102770 0x7f94501115c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:46.349 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.519+0000 7f9456663700 1 -- 172.21.15.6:0/1951139128 >> 172.21.15.6:0/1951139128 conn(0x7f94500fded0 msgr2=0x7f94500ff5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:46.349 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.519+0000 7f9456663700 1 -- 172.21.15.6:0/1951139128 shutdown_connections 2024-02-09T21:50:46.349 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:45.519+0000 7f9456663700 1 -- 172.21.15.6:0/1951139128 wait complete. 2024-02-09T21:50:46.353 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-09T21:50:46.353 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout disabled 2024-02-09T21:50:46.361 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-09T21:50:46.361 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout inactive 2024-02-09T21:50:46.361 INFO:teuthology.orchestra.run.smithi006.stdout:firewalld.service is not enabled 2024-02-09T21:50:46.361 INFO:teuthology.orchestra.run.smithi006.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-09T21:50:46.363 INFO:teuthology.orchestra.run.smithi006.stdout:Ceph Dashboard is now available at: 2024-02-09T21:50:46.363 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:50:46.363 INFO:teuthology.orchestra.run.smithi006.stdout: URL: https://smithi006.front.sepia.ceph.com:8443/ 2024-02-09T21:50:46.363 INFO:teuthology.orchestra.run.smithi006.stdout: User: admin 2024-02-09T21:50:46.363 INFO:teuthology.orchestra.run.smithi006.stdout: Password: 70ec49xuk3 2024-02-09T21:50:46.363 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:50:46.363 INFO:teuthology.orchestra.run.smithi006.stdout:Saving cluster configuration to /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/config directory 2024-02-09T21:50:46.364 INFO:teuthology.orchestra.run.smithi006.stdout:Enabling autotune for osd_memory_target 2024-02-09T21:50:47.605 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.772+0000 7fa10e459700 1 Processor -- start 2024-02-09T21:50:47.605 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.772+0000 7fa10e459700 1 -- start start 2024-02-09T21:50:47.605 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.772+0000 7fa10e459700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa108076ba0 0x7fa108105400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:47.605 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.772+0000 7fa10e459700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa108105940 con 0x7fa108076ba0 2024-02-09T21:50:47.605 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.772+0000 7fa107fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa108076ba0 0x7fa108105400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:47.605 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.772+0000 7fa107fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa108076ba0 0x7fa108105400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50590/0 (socket says 172.21.15.6:50590) 2024-02-09T21:50:47.605 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.772+0000 7fa107fff700 1 -- 172.21.15.6:0/2351949731 learned_addr learned my addr 172.21.15.6:0/2351949731 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:47.605 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.772+0000 7fa107fff700 1 -- 172.21.15.6:0/2351949731 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa108105a80 con 0x7fa108076ba0 2024-02-09T21:50:47.605 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.773+0000 7fa107fff700 1 --2- 172.21.15.6:0/2351949731 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa108076ba0 0x7fa108105400 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa0f80098e0 tx=0x7fa0f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0f7143376c5a8a2 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:47.606 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.773+0000 7fa106ffd700 1 -- 172.21.15.6:0/2351949731 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0f800b8f0 con 0x7fa108076ba0 2024-02-09T21:50:47.606 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.773+0000 7fa106ffd700 1 -- 172.21.15.6:0/2351949731 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa0f8011420 con 0x7fa108076ba0 2024-02-09T21:50:47.606 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.773+0000 7fa106ffd700 1 -- 172.21.15.6:0/2351949731 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0f8010540 con 0x7fa108076ba0 2024-02-09T21:50:47.606 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.774+0000 7fa10e459700 1 -- 172.21.15.6:0/2351949731 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa108076ba0 msgr2=0x7fa108105400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:47.606 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.774+0000 7fa10e459700 1 --2- 172.21.15.6:0/2351949731 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa108076ba0 0x7fa108105400 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa0f80098e0 tx=0x7fa0f8009250 comp rx=0 tx=0).stop 2024-02-09T21:50:47.606 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.774+0000 7fa10e459700 1 -- 172.21.15.6:0/2351949731 shutdown_connections 2024-02-09T21:50:47.606 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.774+0000 7fa10e459700 1 --2- 172.21.15.6:0/2351949731 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa108076ba0 0x7fa108105400 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:47.606 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.774+0000 7fa10e459700 1 -- 172.21.15.6:0/2351949731 >> 172.21.15.6:0/2351949731 conn(0x7fa1080fcdd0 msgr2=0x7fa1080ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:47.606 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.774+0000 7fa10e459700 1 -- 172.21.15.6:0/2351949731 shutdown_connections 2024-02-09T21:50:47.606 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.774+0000 7fa10e459700 1 -- 172.21.15.6:0/2351949731 wait complete. 2024-02-09T21:50:47.606 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.774+0000 7fa10e459700 1 Processor -- start 2024-02-09T21:50:47.606 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.774+0000 7fa10e459700 1 -- start start 2024-02-09T21:50:47.606 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.775+0000 7fa10e459700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa108076ba0 0x7fa108111590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:47.607 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.775+0000 7fa10e459700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa108111ad0 con 0x7fa108076ba0 2024-02-09T21:50:47.607 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.775+0000 7fa107fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa108076ba0 0x7fa108111590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:47.607 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.775+0000 7fa107fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa108076ba0 0x7fa108111590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50606/0 (socket says 172.21.15.6:50606) 2024-02-09T21:50:47.607 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.775+0000 7fa107fff700 1 -- 172.21.15.6:0/2028721137 learned_addr learned my addr 172.21.15.6:0/2028721137 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:47.607 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.775+0000 7fa107fff700 1 -- 172.21.15.6:0/2028721137 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f80094f0 con 0x7fa108076ba0 2024-02-09T21:50:47.607 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.775+0000 7fa107fff700 1 --2- 172.21.15.6:0/2028721137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa108076ba0 0x7fa108111590 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8011cf0 tx=0x7fa0f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:47.607 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.775+0000 7fa104ff9700 1 -- 172.21.15.6:0/2028721137 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0f8010800 con 0x7fa108076ba0 2024-02-09T21:50:47.607 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.775+0000 7fa10e459700 1 -- 172.21.15.6:0/2028721137 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa108111cd0 con 0x7fa108076ba0 2024-02-09T21:50:47.607 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.775+0000 7fa104ff9700 1 -- 172.21.15.6:0/2028721137 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa0f8010e30 con 0x7fa108076ba0 2024-02-09T21:50:47.607 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.775+0000 7fa10e459700 1 -- 172.21.15.6:0/2028721137 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa108112970 con 0x7fa108076ba0 2024-02-09T21:50:47.607 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.775+0000 7fa104ff9700 1 -- 172.21.15.6:0/2028721137 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0f801ae80 con 0x7fa108076ba0 2024-02-09T21:50:47.607 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.776+0000 7fa104ff9700 1 -- 172.21.15.6:0/2028721137 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fa0f802d3f0 con 0x7fa108076ba0 2024-02-09T21:50:47.607 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.776+0000 7fa10e459700 1 -- 172.21.15.6:0/2028721137 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0f40052e0 con 0x7fa108076ba0 2024-02-09T21:50:47.607 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.777+0000 7fa104ff9700 1 --2- 172.21.15.6:0/2028721137 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fa0f0036830 0x7fa0f0038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.777+0000 7fa104ff9700 1 -- 172.21.15.6:0/2028721137 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa0f804a280 con 0x7fa108076ba0 2024-02-09T21:50:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.777+0000 7fa1077fe700 1 --2- 172.21.15.6:0/2028721137 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fa0f0036830 0x7fa0f0038ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.777+0000 7fa1077fe700 1 --2- 172.21.15.6:0/2028721137 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fa0f0036830 0x7fa0f0038ce0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc009990 tx=0x7fa0fc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.785+0000 7fa104ff9700 1 -- 172.21.15.6:0/2028721137 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa0f802b8e0 con 0x7fa108076ba0 2024-02-09T21:50:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.979+0000 7fa10e459700 1 -- 172.21.15.6:0/2028721137 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fa0f4005b10 con 0x7fa108076ba0 2024-02-09T21:50:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.979+0000 7fa104ff9700 1 -- 172.21.15.6:0/2028721137 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fa0f802c350 con 0x7fa108076ba0 2024-02-09T21:50:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.981+0000 7fa10e459700 1 -- 172.21.15.6:0/2028721137 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fa0f0036830 msgr2=0x7fa0f0038ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.981+0000 7fa10e459700 1 --2- 172.21.15.6:0/2028721137 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fa0f0036830 0x7fa0f0038ce0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc009990 tx=0x7fa0fc006e40 comp rx=0 tx=0).stop 2024-02-09T21:50:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.981+0000 7fa10e459700 1 -- 172.21.15.6:0/2028721137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa108076ba0 msgr2=0x7fa108111590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.981+0000 7fa10e459700 1 --2- 172.21.15.6:0/2028721137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa108076ba0 0x7fa108111590 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8011cf0 tx=0x7fa0f8009250 comp rx=0 tx=0).stop 2024-02-09T21:50:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.981+0000 7fa10e459700 1 -- 172.21.15.6:0/2028721137 shutdown_connections 2024-02-09T21:50:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.981+0000 7fa10e459700 1 --2- 172.21.15.6:0/2028721137 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fa0f0036830 0x7fa0f0038ce0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.981+0000 7fa10e459700 1 --2- 172.21.15.6:0/2028721137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa108076ba0 0x7fa108111590 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:47.609 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.981+0000 7fa10e459700 1 -- 172.21.15.6:0/2028721137 >> 172.21.15.6:0/2028721137 conn(0x7fa1080fcdd0 msgr2=0x7fa1080fe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:47.609 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.982+0000 7fa10e459700 1 -- 172.21.15.6:0/2028721137 shutdown_connections 2024-02-09T21:50:47.609 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:46.982+0000 7fa10e459700 1 -- 172.21.15.6:0/2028721137 wait complete. 2024-02-09T21:50:48.602 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:48 smithi006 ceph-mon[29258]: mgrmap e13: smithi006.glirxt(active, since 6s) 2024-02-09T21:50:48.884 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.975+0000 7fe007a41700 1 Processor -- start 2024-02-09T21:50:48.884 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.976+0000 7fe007a41700 1 -- start start 2024-02-09T21:50:48.884 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.976+0000 7fe007a41700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe000103e40 0x7fe000104250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:48.884 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.976+0000 7fe007a41700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe000104790 con 0x7fe000103e40 2024-02-09T21:50:48.884 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.976+0000 7fe0057dd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe000103e40 0x7fe000104250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:48.885 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.976+0000 7fe0057dd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe000103e40 0x7fe000104250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34134/0 (socket says 172.21.15.6:34134) 2024-02-09T21:50:48.885 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.976+0000 7fe0057dd700 1 -- 172.21.15.6:0/1101488394 learned_addr learned my addr 172.21.15.6:0/1101488394 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:48.885 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.976+0000 7fe0057dd700 1 -- 172.21.15.6:0/1101488394 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0001048d0 con 0x7fe000103e40 2024-02-09T21:50:48.885 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.976+0000 7fe0057dd700 1 --2- 172.21.15.6:0/1101488394 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe000103e40 0x7fe000104250 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdffc0098e0 tx=0x7fdffc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebbe464c82fd3b90 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:48.885 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.976+0000 7fdff7fff700 1 -- 172.21.15.6:0/1101488394 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdffc00b860 con 0x7fe000103e40 2024-02-09T21:50:48.885 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.977+0000 7fdff7fff700 1 -- 172.21.15.6:0/1101488394 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdffc00be90 con 0x7fe000103e40 2024-02-09T21:50:48.885 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.977+0000 7fdff7fff700 1 -- 172.21.15.6:0/1101488394 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdffc0103c0 con 0x7fe000103e40 2024-02-09T21:50:48.885 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.977+0000 7fe007a41700 1 -- 172.21.15.6:0/1101488394 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe000103e40 msgr2=0x7fe000104250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:48.885 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.977+0000 7fe007a41700 1 --2- 172.21.15.6:0/1101488394 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe000103e40 0x7fe000104250 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdffc0098e0 tx=0x7fdffc009250 comp rx=0 tx=0).stop 2024-02-09T21:50:48.885 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.977+0000 7fe007a41700 1 -- 172.21.15.6:0/1101488394 shutdown_connections 2024-02-09T21:50:48.885 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.977+0000 7fe007a41700 1 --2- 172.21.15.6:0/1101488394 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe000103e40 0x7fe000104250 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:48.885 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.977+0000 7fe007a41700 1 -- 172.21.15.6:0/1101488394 >> 172.21.15.6:0/1101488394 conn(0x7fe0000fc580 msgr2=0x7fe0000fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:48.885 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.977+0000 7fe007a41700 1 -- 172.21.15.6:0/1101488394 shutdown_connections 2024-02-09T21:50:48.885 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.977+0000 7fe007a41700 1 -- 172.21.15.6:0/1101488394 wait complete. 2024-02-09T21:50:48.885 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.978+0000 7fe007a41700 1 Processor -- start 2024-02-09T21:50:48.886 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.978+0000 7fe007a41700 1 -- start start 2024-02-09T21:50:48.886 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.978+0000 7fe007a41700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe000103e40 0x7fe000111860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:48.886 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.978+0000 7fe007a41700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe000111da0 con 0x7fe000103e40 2024-02-09T21:50:48.886 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.978+0000 7fe0057dd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe000103e40 0x7fe000111860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:48.886 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.978+0000 7fe0057dd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe000103e40 0x7fe000111860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34150/0 (socket says 172.21.15.6:34150) 2024-02-09T21:50:48.886 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.978+0000 7fe0057dd700 1 -- 172.21.15.6:0/218648324 learned_addr learned my addr 172.21.15.6:0/218648324 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:48.886 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.978+0000 7fe0057dd700 1 -- 172.21.15.6:0/218648324 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdffc0094f0 con 0x7fe000103e40 2024-02-09T21:50:48.886 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.978+0000 7fe0057dd700 1 --2- 172.21.15.6:0/218648324 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe000103e40 0x7fe000111860 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdffc011650 tx=0x7fdffc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:48.886 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.979+0000 7fdff5ffb700 1 -- 172.21.15.6:0/218648324 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdffc010890 con 0x7fe000103e40 2024-02-09T21:50:48.886 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.979+0000 7fdff5ffb700 1 -- 172.21.15.6:0/218648324 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdffc010ec0 con 0x7fe000103e40 2024-02-09T21:50:48.886 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.979+0000 7fe007a41700 1 -- 172.21.15.6:0/218648324 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe000111fa0 con 0x7fe000103e40 2024-02-09T21:50:48.886 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.979+0000 7fdff5ffb700 1 -- 172.21.15.6:0/218648324 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdffc01a430 con 0x7fe000103e40 2024-02-09T21:50:48.886 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.979+0000 7fe007a41700 1 -- 172.21.15.6:0/218648324 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe000112c40 con 0x7fe000103e40 2024-02-09T21:50:48.886 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.980+0000 7fdff5ffb700 1 -- 172.21.15.6:0/218648324 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7fdffc01a6b0 con 0x7fe000103e40 2024-02-09T21:50:48.886 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.980+0000 7fdff5ffb700 1 --2- 172.21.15.6:0/218648324 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fdfec036880 0x7fdfec038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:48.887 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.980+0000 7fdff5ffb700 1 -- 172.21.15.6:0/218648324 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fdffc04a730 con 0x7fe000103e40 2024-02-09T21:50:48.887 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.980+0000 7fe007a41700 1 -- 172.21.15.6:0/218648324 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe00005f5c0 con 0x7fe000103e40 2024-02-09T21:50:48.887 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.980+0000 7fe004fdc700 1 --2- 172.21.15.6:0/218648324 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fdfec036880 0x7fdfec038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:48.887 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.980+0000 7fe004fdc700 1 --2- 172.21.15.6:0/218648324 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fdfec036880 0x7fdfec038d30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdff0009990 tx=0x7fdff0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:48.887 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:47.985+0000 7fdff5ffb700 1 -- 172.21.15.6:0/218648324 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fdffc02d3f0 con 0x7fe000103e40 2024-02-09T21:50:48.887 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:48.287+0000 7fe007a41700 1 -- 172.21.15.6:0/218648324 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fe000064ad0 con 0x7fe000103e40 2024-02-09T21:50:48.887 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:48.291+0000 7fdff5ffb700 1 -- 172.21.15.6:0/218648324 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7fdffc018350 con 0x7fe000103e40 2024-02-09T21:50:48.887 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:48.293+0000 7fe007a41700 1 -- 172.21.15.6:0/218648324 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fdfec036880 msgr2=0x7fdfec038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:48.887 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:48.293+0000 7fe007a41700 1 --2- 172.21.15.6:0/218648324 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fdfec036880 0x7fdfec038d30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdff0009990 tx=0x7fdff0006e40 comp rx=0 tx=0).stop 2024-02-09T21:50:48.887 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:48.293+0000 7fe007a41700 1 -- 172.21.15.6:0/218648324 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe000103e40 msgr2=0x7fe000111860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:48.887 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:48.293+0000 7fe007a41700 1 --2- 172.21.15.6:0/218648324 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe000103e40 0x7fe000111860 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdffc011650 tx=0x7fdffc009250 comp rx=0 tx=0).stop 2024-02-09T21:50:48.887 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:48.293+0000 7fe007a41700 1 -- 172.21.15.6:0/218648324 shutdown_connections 2024-02-09T21:50:48.888 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:48.293+0000 7fe007a41700 1 --2- 172.21.15.6:0/218648324 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fdfec036880 0x7fdfec038d30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:48.888 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:48.293+0000 7fe007a41700 1 --2- 172.21.15.6:0/218648324 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe000103e40 0x7fe000111860 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:48.888 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:48.293+0000 7fe007a41700 1 -- 172.21.15.6:0/218648324 >> 172.21.15.6:0/218648324 conn(0x7fe0000fc580 msgr2=0x7fe0000fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:48.888 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:48.294+0000 7fe007a41700 1 -- 172.21.15.6:0/218648324 shutdown_connections 2024-02-09T21:50:48.888 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr 2024-02-09T21:50:48.294+0000 7fe007a41700 1 -- 172.21.15.6:0/218648324 wait complete. 2024-02-09T21:50:48.888 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-02-09T21:50:48.888 INFO:teuthology.orchestra.run.smithi006.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-02-09T21:50:48.888 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:50:48.888 INFO:teuthology.orchestra.run.smithi006.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid ee26d820-c794-11ee-95b8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-09T21:50:48.888 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:50:48.888 INFO:teuthology.orchestra.run.smithi006.stdout:Or, if you are only running a single cluster on this host: 2024-02-09T21:50:48.888 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:50:48.888 INFO:teuthology.orchestra.run.smithi006.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-02-09T21:50:48.888 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:50:48.888 INFO:teuthology.orchestra.run.smithi006.stdout:Please consider enabling telemetry to help improve Ceph: 2024-02-09T21:50:48.888 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:50:48.889 INFO:teuthology.orchestra.run.smithi006.stdout: ceph telemetry on 2024-02-09T21:50:48.889 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:50:48.889 INFO:teuthology.orchestra.run.smithi006.stdout:For more information see: 2024-02-09T21:50:48.889 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:50:48.889 INFO:teuthology.orchestra.run.smithi006.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-02-09T21:50:48.889 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:50:48.889 INFO:teuthology.orchestra.run.smithi006.stdout:Bootstrap complete. 2024-02-09T21:50:48.922 INFO:tasks.cephadm:Fetching config... 2024-02-09T21:50:48.922 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-02-09T21:50:48.923 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-09T21:50:48.979 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-09T21:50:48.980 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-02-09T21:50:48.980 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-09T21:50:49.034 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-09T21:50:49.035 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-02-09T21:50:49.035 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/keyring of=/dev/stdout 2024-02-09T21:50:49.102 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-09T21:50:49.103 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-02-09T21:50:49.103 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-09T21:50:49.158 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-09T21:50:49.158 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtYl/zJGGbFU4MjGUOF3yv3UUfjQxWFnzXGwhAabHxgwiZ4UbFBrCadqOsyTodexdSn+Ufxd8t60A6iqL+2so01tSSxAqr2xXw61tjkK1Q40+WnPI+kFHk/OvMtKHPy20Yl+jX9WUwd4CwcelzBrnmaZpR/pXvawnM2OWFkDRaoyJA3gK0m8YrHFdlvAFTwD8dsn79jGu9+xilARVrnNyBYGUwgeLQNFoGdlDanZHz8PX3q3XA/Tfl0fCRA/gTr8N5d4sYS7AMvEZY5cCBxXB/ZH8iYUltyPZg59/quEM3XPw8EcBzgl/ZmfPAncNDZWGftptFzPeb54i7qQOUc3Q9vITY7OnxGf36qeXJbpP5eGEOUlCfpbIREhiBJDKcjG9pBaxDrEPfVsEMbGC6FTrUVnI8TqyzyCCfmqQomJMpDr5TPtwq5GaYbtkz+NBLSVXTdqXDPRSqfvEX4U2EhDF6oTIf63fXyn+jUMdyk9O00LMzlUHE8Y6IVIx9wXzLO10= ceph-ee26d820-c794-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-09T21:50:49.238 INFO:teuthology.orchestra.run.smithi006.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtYl/zJGGbFU4MjGUOF3yv3UUfjQxWFnzXGwhAabHxgwiZ4UbFBrCadqOsyTodexdSn+Ufxd8t60A6iqL+2so01tSSxAqr2xXw61tjkK1Q40+WnPI+kFHk/OvMtKHPy20Yl+jX9WUwd4CwcelzBrnmaZpR/pXvawnM2OWFkDRaoyJA3gK0m8YrHFdlvAFTwD8dsn79jGu9+xilARVrnNyBYGUwgeLQNFoGdlDanZHz8PX3q3XA/Tfl0fCRA/gTr8N5d4sYS7AMvEZY5cCBxXB/ZH8iYUltyPZg59/quEM3XPw8EcBzgl/ZmfPAncNDZWGftptFzPeb54i7qQOUc3Q9vITY7OnxGf36qeXJbpP5eGEOUlCfpbIREhiBJDKcjG9pBaxDrEPfVsEMbGC6FTrUVnI8TqyzyCCfmqQomJMpDr5TPtwq5GaYbtkz+NBLSVXTdqXDPRSqfvEX4U2EhDF6oTIf63fXyn+jUMdyk9O00LMzlUHE8Y6IVIx9wXzLO10= ceph-ee26d820-c794-11ee-95b8-87774f69a715 2024-02-09T21:50:49.252 DEBUG:teuthology.orchestra.run.smithi090:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtYl/zJGGbFU4MjGUOF3yv3UUfjQxWFnzXGwhAabHxgwiZ4UbFBrCadqOsyTodexdSn+Ufxd8t60A6iqL+2so01tSSxAqr2xXw61tjkK1Q40+WnPI+kFHk/OvMtKHPy20Yl+jX9WUwd4CwcelzBrnmaZpR/pXvawnM2OWFkDRaoyJA3gK0m8YrHFdlvAFTwD8dsn79jGu9+xilARVrnNyBYGUwgeLQNFoGdlDanZHz8PX3q3XA/Tfl0fCRA/gTr8N5d4sYS7AMvEZY5cCBxXB/ZH8iYUltyPZg59/quEM3XPw8EcBzgl/ZmfPAncNDZWGftptFzPeb54i7qQOUc3Q9vITY7OnxGf36qeXJbpP5eGEOUlCfpbIREhiBJDKcjG9pBaxDrEPfVsEMbGC6FTrUVnI8TqyzyCCfmqQomJMpDr5TPtwq5GaYbtkz+NBLSVXTdqXDPRSqfvEX4U2EhDF6oTIf63fXyn+jUMdyk9O00LMzlUHE8Y6IVIx9wXzLO10= ceph-ee26d820-c794-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-09T21:50:49.299 INFO:teuthology.orchestra.run.smithi090.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtYl/zJGGbFU4MjGUOF3yv3UUfjQxWFnzXGwhAabHxgwiZ4UbFBrCadqOsyTodexdSn+Ufxd8t60A6iqL+2so01tSSxAqr2xXw61tjkK1Q40+WnPI+kFHk/OvMtKHPy20Yl+jX9WUwd4CwcelzBrnmaZpR/pXvawnM2OWFkDRaoyJA3gK0m8YrHFdlvAFTwD8dsn79jGu9+xilARVrnNyBYGUwgeLQNFoGdlDanZHz8PX3q3XA/Tfl0fCRA/gTr8N5d4sYS7AMvEZY5cCBxXB/ZH8iYUltyPZg59/quEM3XPw8EcBzgl/ZmfPAncNDZWGftptFzPeb54i7qQOUc3Q9vITY7OnxGf36qeXJbpP5eGEOUlCfpbIREhiBJDKcjG9pBaxDrEPfVsEMbGC6FTrUVnI8TqyzyCCfmqQomJMpDr5TPtwq5GaYbtkz+NBLSVXTdqXDPRSqfvEX4U2EhDF6oTIf63fXyn+jUMdyk9O00LMzlUHE8Y6IVIx9wXzLO10= ceph-ee26d820-c794-11ee-95b8-87774f69a715 2024-02-09T21:50:49.313 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-09T21:50:49.571 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:50:49.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:49 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/218648324' entity='client.admin' 2024-02-09T21:50:51.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.644+0000 7fe9e9d2d700 1 -- 172.21.15.6:0/2714695262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9e40fbad0 msgr2=0x7fe9e40fbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:51.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.644+0000 7fe9e9d2d700 1 --2- 172.21.15.6:0/2714695262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9e40fbad0 0x7fe9e40fbee0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe9d40048c0 tx=0x7fe9d4009e30 comp rx=0 tx=0).stop 2024-02-09T21:50:51.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.644+0000 7fe9e9d2d700 1 -- 172.21.15.6:0/2714695262 shutdown_connections 2024-02-09T21:50:51.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.644+0000 7fe9e9d2d700 1 --2- 172.21.15.6:0/2714695262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9e40fbad0 0x7fe9e40fbee0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:51.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.644+0000 7fe9e9d2d700 1 -- 172.21.15.6:0/2714695262 >> 172.21.15.6:0/2714695262 conn(0x7fe9e40f7190 msgr2=0x7fe9e40f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:51.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.644+0000 7fe9e9d2d700 1 -- 172.21.15.6:0/2714695262 shutdown_connections 2024-02-09T21:50:51.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.644+0000 7fe9e9d2d700 1 -- 172.21.15.6:0/2714695262 wait complete. 2024-02-09T21:50:51.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.644+0000 7fe9e9d2d700 1 Processor -- start 2024-02-09T21:50:51.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.645+0000 7fe9e9d2d700 1 -- start start 2024-02-09T21:50:51.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.645+0000 7fe9e9d2d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9e40fbad0 0x7fe9e410f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:51.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.645+0000 7fe9e9d2d700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e410f600 con 0x7fe9e40fbad0 2024-02-09T21:50:51.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.645+0000 7fe9e37fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9e40fbad0 0x7fe9e410f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:51.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.645+0000 7fe9e37fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9e40fbad0 0x7fe9e410f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34160/0 (socket says 172.21.15.6:34160) 2024-02-09T21:50:51.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.645+0000 7fe9e37fe700 1 -- 172.21.15.6:0/1226660290 learned_addr learned my addr 172.21.15.6:0/1226660290 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:51.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.645+0000 7fe9e37fe700 1 -- 172.21.15.6:0/1226660290 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d4005040 con 0x7fe9e40fbad0 2024-02-09T21:50:51.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.645+0000 7fe9e37fe700 1 --2- 172.21.15.6:0/1226660290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9e40fbad0 0x7fe9e410f0c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe9d4009b80 tx=0x7fe9d4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:51.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.646+0000 7fe9cbfff700 1 -- 172.21.15.6:0/1226660290 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9d400b5c0 con 0x7fe9e40fbad0 2024-02-09T21:50:51.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.646+0000 7fe9cbfff700 1 -- 172.21.15.6:0/1226660290 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe9d400be70 con 0x7fe9e40fbad0 2024-02-09T21:50:51.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.646+0000 7fe9e9d2d700 1 -- 172.21.15.6:0/1226660290 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9e410f800 con 0x7fe9e40fbad0 2024-02-09T21:50:51.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.646+0000 7fe9cbfff700 1 -- 172.21.15.6:0/1226660290 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9d401c660 con 0x7fe9e40fbad0 2024-02-09T21:50:51.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.646+0000 7fe9e9d2d700 1 -- 172.21.15.6:0/1226660290 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9e41104a0 con 0x7fe9e40fbad0 2024-02-09T21:50:51.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.647+0000 7fe9cbfff700 1 -- 172.21.15.6:0/1226660290 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7fe9d4013420 con 0x7fe9e40fbad0 2024-02-09T21:50:51.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.647+0000 7fe9e9d2d700 1 -- 172.21.15.6:0/1226660290 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9e405f5c0 con 0x7fe9e40fbad0 2024-02-09T21:50:51.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.647+0000 7fe9cbfff700 1 --2- 172.21.15.6:0/1226660290 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fe9cc036870 0x7fe9cc038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:51.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.647+0000 7fe9cbfff700 1 -- 172.21.15.6:0/1226660290 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe9d40499d0 con 0x7fe9e40fbad0 2024-02-09T21:50:51.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.647+0000 7fe9e2ffd700 1 --2- 172.21.15.6:0/1226660290 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fe9cc036870 0x7fe9cc038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:51.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.647+0000 7fe9e2ffd700 1 --2- 172.21.15.6:0/1226660290 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fe9cc036870 0x7fe9cc038d20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8009990 tx=0x7fe9d8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:51.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.652+0000 7fe9cbfff700 1 -- 172.21.15.6:0/1226660290 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe9d4013b10 con 0x7fe9e40fbad0 2024-02-09T21:50:51.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.848+0000 7fe9e9d2d700 1 -- 172.21.15.6:0/1226660290 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fe9e404bed0 con 0x7fe9e40fbad0 2024-02-09T21:50:51.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.852+0000 7fe9cbfff700 1 -- 172.21.15.6:0/1226660290 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fe9d4011370 con 0x7fe9e40fbad0 2024-02-09T21:50:51.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.854+0000 7fe9e9d2d700 1 -- 172.21.15.6:0/1226660290 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fe9cc036870 msgr2=0x7fe9cc038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:51.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.854+0000 7fe9e9d2d700 1 --2- 172.21.15.6:0/1226660290 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fe9cc036870 0x7fe9cc038d20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8009990 tx=0x7fe9d8006e40 comp rx=0 tx=0).stop 2024-02-09T21:50:51.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.854+0000 7fe9e9d2d700 1 -- 172.21.15.6:0/1226660290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9e40fbad0 msgr2=0x7fe9e410f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:51.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.854+0000 7fe9e9d2d700 1 --2- 172.21.15.6:0/1226660290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9e40fbad0 0x7fe9e410f0c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe9d4009b80 tx=0x7fe9d4009e30 comp rx=0 tx=0).stop 2024-02-09T21:50:51.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.854+0000 7fe9e9d2d700 1 -- 172.21.15.6:0/1226660290 shutdown_connections 2024-02-09T21:50:51.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.854+0000 7fe9e9d2d700 1 --2- 172.21.15.6:0/1226660290 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fe9cc036870 0x7fe9cc038d20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:51.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.854+0000 7fe9e9d2d700 1 --2- 172.21.15.6:0/1226660290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9e40fbad0 0x7fe9e410f0c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:51.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.854+0000 7fe9e9d2d700 1 -- 172.21.15.6:0/1226660290 >> 172.21.15.6:0/1226660290 conn(0x7fe9e40f7190 msgr2=0x7fe9e4065980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:51.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.855+0000 7fe9e9d2d700 1 -- 172.21.15.6:0/1226660290 shutdown_connections 2024-02-09T21:50:51.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:51.855+0000 7fe9e9d2d700 1 -- 172.21.15.6:0/1226660290 wait complete. 2024-02-09T21:50:52.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:51 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:52.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:51 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:52.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:51 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config rm", "who": "osd/host:smithi006", "name": "osd_memory_target"}]: dispatch 2024-02-09T21:50:52.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:51 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:52.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:51 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi006", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-09T21:50:52.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:51 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi006", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-09T21:50:52.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:51 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:50:52.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:51 smithi006 ceph-mon[29258]: Deploying daemon crash.smithi006 on smithi006 2024-02-09T21:50:52.426 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-09T21:50:52.426 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-09T21:50:52.678 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:50:53.123 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:52 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/1226660290' entity='client.admin' 2024-02-09T21:50:55.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.391+0000 7f627c3df700 1 -- 172.21.15.6:0/3273432391 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62740fcbd0 msgr2=0x7f62740fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:55.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.391+0000 7f627c3df700 1 --2- 172.21.15.6:0/3273432391 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62740fcbd0 0x7f62740fcfe0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6264009c60 tx=0x7f6264004d10 comp rx=0 tx=0).stop 2024-02-09T21:50:55.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.391+0000 7f627c3df700 1 -- 172.21.15.6:0/3273432391 shutdown_connections 2024-02-09T21:50:55.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.391+0000 7f627c3df700 1 --2- 172.21.15.6:0/3273432391 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62740fcbd0 0x7f62740fcfe0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:55.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.391+0000 7f627c3df700 1 -- 172.21.15.6:0/3273432391 >> 172.21.15.6:0/3273432391 conn(0x7f62740f8290 msgr2=0x7f62740fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:55.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.391+0000 7f627c3df700 1 -- 172.21.15.6:0/3273432391 shutdown_connections 2024-02-09T21:50:55.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.391+0000 7f627c3df700 1 -- 172.21.15.6:0/3273432391 wait complete. 2024-02-09T21:50:55.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.391+0000 7f627c3df700 1 Processor -- start 2024-02-09T21:50:55.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.392+0000 7f627c3df700 1 -- start start 2024-02-09T21:50:55.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.392+0000 7f627c3df700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62740fcbd0 0x7f627410cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:55.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.392+0000 7f627c3df700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627410d490 con 0x7f62740fcbd0 2024-02-09T21:50:55.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.392+0000 7f627a17b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62740fcbd0 0x7f627410cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:55.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.392+0000 7f627a17b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62740fcbd0 0x7f627410cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34170/0 (socket says 172.21.15.6:34170) 2024-02-09T21:50:55.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.392+0000 7f627a17b700 1 -- 172.21.15.6:0/2520188975 learned_addr learned my addr 172.21.15.6:0/2520188975 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:55.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.392+0000 7f627a17b700 1 -- 172.21.15.6:0/2520188975 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6264009940 con 0x7f62740fcbd0 2024-02-09T21:50:55.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.392+0000 7f627a17b700 1 --2- 172.21.15.6:0/2520188975 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62740fcbd0 0x7f627410cf50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f62640097d0 tx=0x7f6264004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:55.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.392+0000 7f6272ffd700 1 -- 172.21.15.6:0/2520188975 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f626401d440 con 0x7f62740fcbd0 2024-02-09T21:50:55.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.392+0000 7f627c3df700 1 -- 172.21.15.6:0/2520188975 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f627410d690 con 0x7f62740fcbd0 2024-02-09T21:50:55.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.392+0000 7f6272ffd700 1 -- 172.21.15.6:0/2520188975 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f626401da70 con 0x7f62740fcbd0 2024-02-09T21:50:55.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.392+0000 7f6272ffd700 1 -- 172.21.15.6:0/2520188975 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6264012a10 con 0x7f62740fcbd0 2024-02-09T21:50:55.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.392+0000 7f627c3df700 1 -- 172.21.15.6:0/2520188975 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f627410e330 con 0x7f62740fcbd0 2024-02-09T21:50:55.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.393+0000 7f6272ffd700 1 -- 172.21.15.6:0/2520188975 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7f6264019040 con 0x7f62740fcbd0 2024-02-09T21:50:55.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.394+0000 7f6270ff9700 1 -- 172.21.15.6:0/2520188975 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f627405f5c0 con 0x7f62740fcbd0 2024-02-09T21:50:55.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.394+0000 7f6272ffd700 1 --2- 172.21.15.6:0/2520188975 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f625c036870 0x7f625c038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:55.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.394+0000 7f6272ffd700 1 -- 172.21.15.6:0/2520188975 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f626404a500 con 0x7f62740fcbd0 2024-02-09T21:50:55.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.394+0000 7f627997a700 1 --2- 172.21.15.6:0/2520188975 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f625c036870 0x7f625c038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:55.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.394+0000 7f627997a700 1 --2- 172.21.15.6:0/2520188975 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f625c036870 0x7f625c038d20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6268009990 tx=0x7f6268006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:55.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.399+0000 7f6272ffd700 1 -- 172.21.15.6:0/2520188975 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f626402eaf0 con 0x7f62740fcbd0 2024-02-09T21:50:55.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.594+0000 7f6270ff9700 1 -- 172.21.15.6:0/2520188975 --> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f627404bf40 con 0x7f625c036870 2024-02-09T21:50:55.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.597+0000 7f6272ffd700 1 -- 172.21.15.6:0/2520188975 <== mgr.14162 v2:172.21.15.6:6800/3321564517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f627404bf40 con 0x7f625c036870 2024-02-09T21:50:55.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.599+0000 7f6270ff9700 1 -- 172.21.15.6:0/2520188975 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f625c036870 msgr2=0x7f625c038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:55.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.599+0000 7f6270ff9700 1 --2- 172.21.15.6:0/2520188975 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f625c036870 0x7f625c038d20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6268009990 tx=0x7f6268006e40 comp rx=0 tx=0).stop 2024-02-09T21:50:55.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.599+0000 7f6270ff9700 1 -- 172.21.15.6:0/2520188975 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62740fcbd0 msgr2=0x7f627410cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:55.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.599+0000 7f6270ff9700 1 --2- 172.21.15.6:0/2520188975 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62740fcbd0 0x7f627410cf50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f62640097d0 tx=0x7f6264004d10 comp rx=0 tx=0).stop 2024-02-09T21:50:55.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.599+0000 7f6270ff9700 1 -- 172.21.15.6:0/2520188975 shutdown_connections 2024-02-09T21:50:55.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.599+0000 7f6270ff9700 1 --2- 172.21.15.6:0/2520188975 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f625c036870 0x7f625c038d20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:55.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.599+0000 7f6270ff9700 1 --2- 172.21.15.6:0/2520188975 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62740fcbd0 0x7f627410cf50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:55.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.599+0000 7f6270ff9700 1 -- 172.21.15.6:0/2520188975 >> 172.21.15.6:0/2520188975 conn(0x7f62740f8290 msgr2=0x7f62740f9060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:55.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.599+0000 7f6270ff9700 1 -- 172.21.15.6:0/2520188975 shutdown_connections 2024-02-09T21:50:55.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:55.599+0000 7f6270ff9700 1 -- 172.21.15.6:0/2520188975 wait complete. 2024-02-09T21:50:56.381 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi090 2024-02-09T21:50:56.381 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-09T21:50:56.381 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/etc/ceph/ceph.conf 2024-02-09T21:50:56.399 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-09T21:50:56.400 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-09T21:50:56.457 INFO:tasks.cephadm:Adding host smithi090 to orchestrator... 2024-02-09T21:50:56.457 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph orch host add smithi090 2024-02-09T21:50:56.735 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:56 smithi006 ceph-mon[29258]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:50:56.735 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:56 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:56.950 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:50:58.113 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:57 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:58.113 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:57 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:58.113 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:57 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:58.113 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:57 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:50:58.113 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:57 smithi006 ceph-mon[29258]: Deploying daemon node-exporter.smithi006 on smithi006 2024-02-09T21:50:58.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:58.591+0000 7f485f50e700 1 -- 172.21.15.6:0/1635971998 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4858065d90 msgr2=0x7f48580ff7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:50:58.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:58.591+0000 7f485f50e700 1 --2- 172.21.15.6:0/1635971998 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4858065d90 0x7f48580ff7e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f48540048c0 tx=0x7f4854009e30 comp rx=0 tx=0).stop 2024-02-09T21:50:58.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:58.591+0000 7f485f50e700 1 -- 172.21.15.6:0/1635971998 shutdown_connections 2024-02-09T21:50:58.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:58.591+0000 7f485f50e700 1 --2- 172.21.15.6:0/1635971998 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4858065d90 0x7f48580ff7e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:50:58.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:58.591+0000 7f485f50e700 1 -- 172.21.15.6:0/1635971998 >> 172.21.15.6:0/1635971998 conn(0x7f48580f7190 msgr2=0x7f48580f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:50:58.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:58.591+0000 7f485f50e700 1 -- 172.21.15.6:0/1635971998 shutdown_connections 2024-02-09T21:50:58.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:58.591+0000 7f485f50e700 1 -- 172.21.15.6:0/1635971998 wait complete. 2024-02-09T21:50:58.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:58.592+0000 7f485f50e700 1 Processor -- start 2024-02-09T21:50:58.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:58.592+0000 7f485f50e700 1 -- start start 2024-02-09T21:50:58.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:58.592+0000 7f485f50e700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4858065d90 0x7f48580fd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:58.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:58.592+0000 7f485f50e700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48580fb6e0 con 0x7f4858065d90 2024-02-09T21:50:58.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:58.592+0000 7f485d2aa700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4858065d90 0x7f48580fd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:58.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:58.592+0000 7f485d2aa700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4858065d90 0x7f48580fd260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60802/0 (socket says 172.21.15.6:60802) 2024-02-09T21:50:58.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:58.592+0000 7f485d2aa700 1 -- 172.21.15.6:0/2351041555 learned_addr learned my addr 172.21.15.6:0/2351041555 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:50:58.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:58.592+0000 7f485d2aa700 1 -- 172.21.15.6:0/2351041555 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4854005040 con 0x7f4858065d90 2024-02-09T21:50:58.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:58.592+0000 7f485d2aa700 1 --2- 172.21.15.6:0/2351041555 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4858065d90 0x7f48580fd260 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4854009680 tx=0x7f4854009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:58.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:58.592+0000 7f4849ffb700 1 -- 172.21.15.6:0/2351041555 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f485400b5c0 con 0x7f4858065d90 2024-02-09T21:50:58.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:58.592+0000 7f4849ffb700 1 -- 172.21.15.6:0/2351041555 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4854004e60 con 0x7f4858065d90 2024-02-09T21:50:58.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:58.592+0000 7f485f50e700 1 -- 172.21.15.6:0/2351041555 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48580fb8e0 con 0x7f4858065d90 2024-02-09T21:50:58.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:58.592+0000 7f4849ffb700 1 -- 172.21.15.6:0/2351041555 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f485401c590 con 0x7f4858065d90 2024-02-09T21:50:58.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:58.593+0000 7f485f50e700 1 -- 172.21.15.6:0/2351041555 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48580fbdd0 con 0x7f4858065d90 2024-02-09T21:50:58.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:58.593+0000 7f4849ffb700 1 -- 172.21.15.6:0/2351041555 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7f485401c810 con 0x7f4858065d90 2024-02-09T21:50:58.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:58.593+0000 7f4849ffb700 1 --2- 172.21.15.6:0/2351041555 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f4844036870 0x7f4844038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:50:58.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:58.594+0000 7f4849ffb700 1 -- 172.21.15.6:0/2351041555 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4854049bc0 con 0x7f4858065d90 2024-02-09T21:50:58.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:58.594+0000 7f485caa9700 1 --2- 172.21.15.6:0/2351041555 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f4844036870 0x7f4844038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:50:58.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:58.594+0000 7f485f50e700 1 -- 172.21.15.6:0/2351041555 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48581069a0 con 0x7f4858065d90 2024-02-09T21:50:58.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:58.594+0000 7f485caa9700 1 --2- 172.21.15.6:0/2351041555 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f4844036870 0x7f4844038d20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f484c009990 tx=0x7f484c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:50:58.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:58.599+0000 7f4849ffb700 1 -- 172.21.15.6:0/2351041555 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4854023300 con 0x7f4858065d90 2024-02-09T21:50:58.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:50:58.810+0000 7f485f50e700 1 -- 172.21.15.6:0/2351041555 --> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi090", "target": ["mon-mgr", ""]}) v1 -- 0x7f48580fc010 con 0x7f4844036870 2024-02-09T21:51:00.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:59 smithi006 ceph-mon[29258]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi090", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:51:00.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:50:59 smithi006 ceph-mon[29258]: Deploying cephadm binary to smithi090 2024-02-09T21:51:00.950 INFO:teuthology.orchestra.run.smithi006.stdout:Added host 'smithi090' with addr '172.21.15.90' 2024-02-09T21:51:00.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:00.946+0000 7f4849ffb700 1 -- 172.21.15.6:0/2351041555 <== mgr.14162 v2:172.21.15.6:6800/3321564517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f48580fc010 con 0x7f4844036870 2024-02-09T21:51:00.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:00.949+0000 7f485f50e700 1 -- 172.21.15.6:0/2351041555 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f4844036870 msgr2=0x7f4844038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:00.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:00.949+0000 7f485f50e700 1 --2- 172.21.15.6:0/2351041555 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f4844036870 0x7f4844038d20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f484c009990 tx=0x7f484c006e40 comp rx=0 tx=0).stop 2024-02-09T21:51:00.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:00.949+0000 7f485f50e700 1 -- 172.21.15.6:0/2351041555 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4858065d90 msgr2=0x7f48580fd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:00.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:00.949+0000 7f485f50e700 1 --2- 172.21.15.6:0/2351041555 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4858065d90 0x7f48580fd260 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4854009680 tx=0x7f4854009e30 comp rx=0 tx=0).stop 2024-02-09T21:51:00.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:00.949+0000 7f485f50e700 1 -- 172.21.15.6:0/2351041555 shutdown_connections 2024-02-09T21:51:00.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:00.949+0000 7f485f50e700 1 --2- 172.21.15.6:0/2351041555 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f4844036870 0x7f4844038d20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:00.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:00.949+0000 7f485f50e700 1 --2- 172.21.15.6:0/2351041555 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4858065d90 0x7f48580fd260 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:00.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:00.949+0000 7f485f50e700 1 -- 172.21.15.6:0/2351041555 >> 172.21.15.6:0/2351041555 conn(0x7f48580f7190 msgr2=0x7f48580f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:00.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:00.949+0000 7f485f50e700 1 -- 172.21.15.6:0/2351041555 shutdown_connections 2024-02-09T21:51:00.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:00.949+0000 7f485f50e700 1 -- 172.21.15.6:0/2351041555 wait complete. 2024-02-09T21:51:01.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:01 smithi006 ceph-mon[29258]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:51:01.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:01 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:51:01.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:01 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:51:02.016 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph orch host ls --format=json 2024-02-09T21:51:02.288 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:51:02.607 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:02 smithi006 ceph-mon[29258]: Added host smithi090 2024-02-09T21:51:02.607 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:02 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:51:02.607 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:02 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:51:02.607 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:02 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:51:02.607 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:02 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:51:02.607 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:02 smithi006 ceph-mon[29258]: Deploying daemon alertmanager.smithi006 on smithi006 2024-02-09T21:51:03.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:03 smithi006 ceph-mon[29258]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:51:04.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.235+0000 7fe81eb92700 1 -- 172.21.15.6:0/205075670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe818065d90 msgr2=0x7fe8180ff7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:04.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.235+0000 7fe81eb92700 1 --2- 172.21.15.6:0/205075670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe818065d90 0x7fe8180ff7e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe80c0048c0 tx=0x7fe80c009e30 comp rx=0 tx=0).stop 2024-02-09T21:51:04.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.235+0000 7fe81eb92700 1 -- 172.21.15.6:0/205075670 shutdown_connections 2024-02-09T21:51:04.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.235+0000 7fe81eb92700 1 --2- 172.21.15.6:0/205075670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe818065d90 0x7fe8180ff7e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:04.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.235+0000 7fe81eb92700 1 -- 172.21.15.6:0/205075670 >> 172.21.15.6:0/205075670 conn(0x7fe8180f7190 msgr2=0x7fe8180f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:04.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.235+0000 7fe81eb92700 1 -- 172.21.15.6:0/205075670 shutdown_connections 2024-02-09T21:51:04.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.235+0000 7fe81eb92700 1 -- 172.21.15.6:0/205075670 wait complete. 2024-02-09T21:51:04.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.235+0000 7fe81eb92700 1 Processor -- start 2024-02-09T21:51:04.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.235+0000 7fe81eb92700 1 -- start start 2024-02-09T21:51:04.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.236+0000 7fe81eb92700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe818065d90 0x7fe8180fd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:04.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.236+0000 7fe81eb92700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8180fb6e0 con 0x7fe818065d90 2024-02-09T21:51:04.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.236+0000 7fe81c92e700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe818065d90 0x7fe8180fd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:51:04.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.236+0000 7fe81c92e700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe818065d90 0x7fe8180fd260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60814/0 (socket says 172.21.15.6:60814) 2024-02-09T21:51:04.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.236+0000 7fe81c92e700 1 -- 172.21.15.6:0/3039709534 learned_addr learned my addr 172.21.15.6:0/3039709534 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:51:04.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.236+0000 7fe81c92e700 1 -- 172.21.15.6:0/3039709534 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe80c005040 con 0x7fe818065d90 2024-02-09T21:51:04.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.236+0000 7fe81c92e700 1 --2- 172.21.15.6:0/3039709534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe818065d90 0x7fe8180fd260 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe80c009b80 tx=0x7fe80c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:51:04.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.236+0000 7fe8157fa700 1 -- 172.21.15.6:0/3039709534 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe80c00b5c0 con 0x7fe818065d90 2024-02-09T21:51:04.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.236+0000 7fe8157fa700 1 -- 172.21.15.6:0/3039709534 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe80c00be70 con 0x7fe818065d90 2024-02-09T21:51:04.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.236+0000 7fe81eb92700 1 -- 172.21.15.6:0/3039709534 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8180fb8e0 con 0x7fe818065d90 2024-02-09T21:51:04.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.236+0000 7fe8157fa700 1 -- 172.21.15.6:0/3039709534 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe80c01c660 con 0x7fe818065d90 2024-02-09T21:51:04.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.236+0000 7fe81eb92700 1 -- 172.21.15.6:0/3039709534 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8180fbdd0 con 0x7fe818065d90 2024-02-09T21:51:04.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.237+0000 7fe8157fa700 1 -- 172.21.15.6:0/3039709534 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7fe80c01c8d0 con 0x7fe818065d90 2024-02-09T21:51:04.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.237+0000 7fe81eb92700 1 -- 172.21.15.6:0/3039709534 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8181069a0 con 0x7fe818065d90 2024-02-09T21:51:04.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.237+0000 7fe8157fa700 1 --2- 172.21.15.6:0/3039709534 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fe808036870 0x7fe808038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:04.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.237+0000 7fe8157fa700 1 -- 172.21.15.6:0/3039709534 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe80c049af0 con 0x7fe818065d90 2024-02-09T21:51:04.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.238+0000 7fe817fff700 1 --2- 172.21.15.6:0/3039709534 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fe808036870 0x7fe808038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:51:04.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.238+0000 7fe817fff700 1 --2- 172.21.15.6:0/3039709534 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fe808036870 0x7fe808038d20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe804009990 tx=0x7fe804006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:51:04.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.242+0000 7fe8157fa700 1 -- 172.21.15.6:0/3039709534 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe80c011dd0 con 0x7fe818065d90 2024-02-09T21:51:04.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.448+0000 7fe81eb92700 1 -- 172.21.15.6:0/3039709534 --> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe81802b3b0 con 0x7fe808036870 2024-02-09T21:51:04.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.449+0000 7fe8157fa700 1 -- 172.21.15.6:0/3039709534 <== mgr.14162 v2:172.21.15.6:6800/3321564517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+158 (secure 0 0 0) 0x7fe81802b3b0 con 0x7fe808036870 2024-02-09T21:51:04.454 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:51:04.454 INFO:teuthology.orchestra.run.smithi006.stdout:[{"addr": "172.21.15.6", "hostname": "smithi006", "labels": [], "status": ""}, {"addr": "172.21.15.90", "hostname": "smithi090", "labels": [], "status": ""}] 2024-02-09T21:51:04.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.451+0000 7fe81eb92700 1 -- 172.21.15.6:0/3039709534 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fe808036870 msgr2=0x7fe808038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:04.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.451+0000 7fe81eb92700 1 --2- 172.21.15.6:0/3039709534 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fe808036870 0x7fe808038d20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe804009990 tx=0x7fe804006e40 comp rx=0 tx=0).stop 2024-02-09T21:51:04.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.451+0000 7fe81eb92700 1 -- 172.21.15.6:0/3039709534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe818065d90 msgr2=0x7fe8180fd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:04.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.451+0000 7fe81eb92700 1 --2- 172.21.15.6:0/3039709534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe818065d90 0x7fe8180fd260 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe80c009b80 tx=0x7fe80c009e30 comp rx=0 tx=0).stop 2024-02-09T21:51:04.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.451+0000 7fe81eb92700 1 -- 172.21.15.6:0/3039709534 shutdown_connections 2024-02-09T21:51:04.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.451+0000 7fe81eb92700 1 --2- 172.21.15.6:0/3039709534 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fe808036870 0x7fe808038d20 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:04.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.451+0000 7fe81eb92700 1 --2- 172.21.15.6:0/3039709534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe818065d90 0x7fe8180fd260 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:04.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.451+0000 7fe81eb92700 1 -- 172.21.15.6:0/3039709534 >> 172.21.15.6:0/3039709534 conn(0x7fe8180f7190 msgr2=0x7fe8180f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:04.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.451+0000 7fe81eb92700 1 -- 172.21.15.6:0/3039709534 shutdown_connections 2024-02-09T21:51:04.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:04.451+0000 7fe81eb92700 1 -- 172.21.15.6:0/3039709534 wait complete. 2024-02-09T21:51:05.558 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-09T21:51:05.558 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph osd crush tunables default 2024-02-09T21:51:05.659 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:05 smithi006 ceph-mon[29258]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:51:05.660 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:05 smithi006 ceph-mon[29258]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T21:51:05.660 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:05 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:51:06.271 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:51:07.546 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:07 smithi006 ceph-mon[29258]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:51:09.580 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:09 smithi006 ceph-mon[29258]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:51:10.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:10.516+0000 7fd6addfe700 1 -- 172.21.15.6:0/3180925929 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6a80fbad0 msgr2=0x7fd6a80fbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:10.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:10.516+0000 7fd6addfe700 1 --2- 172.21.15.6:0/3180925929 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6a80fbad0 0x7fd6a80fbee0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd6980048c0 tx=0x7fd698009e30 comp rx=0 tx=0).stop 2024-02-09T21:51:10.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:10.516+0000 7fd6addfe700 1 -- 172.21.15.6:0/3180925929 shutdown_connections 2024-02-09T21:51:10.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:10.516+0000 7fd6addfe700 1 --2- 172.21.15.6:0/3180925929 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6a80fbad0 0x7fd6a80fbee0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:10.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:10.516+0000 7fd6addfe700 1 -- 172.21.15.6:0/3180925929 >> 172.21.15.6:0/3180925929 conn(0x7fd6a80f7190 msgr2=0x7fd6a80f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:10.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:10.516+0000 7fd6addfe700 1 -- 172.21.15.6:0/3180925929 shutdown_connections 2024-02-09T21:51:10.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:10.516+0000 7fd6addfe700 1 -- 172.21.15.6:0/3180925929 wait complete. 2024-02-09T21:51:10.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:10.517+0000 7fd6addfe700 1 Processor -- start 2024-02-09T21:51:10.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:10.517+0000 7fd6addfe700 1 -- start start 2024-02-09T21:51:10.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:10.517+0000 7fd6addfe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6a80fbad0 0x7fd6a810cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:10.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:10.517+0000 7fd6addfe700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a810d400 con 0x7fd6a80fbad0 2024-02-09T21:51:10.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:10.517+0000 7fd6a77fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6a80fbad0 0x7fd6a810cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:51:10.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:10.517+0000 7fd6a77fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6a80fbad0 0x7fd6a810cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39442/0 (socket says 172.21.15.6:39442) 2024-02-09T21:51:10.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:10.517+0000 7fd6a77fe700 1 -- 172.21.15.6:0/2563589583 learned_addr learned my addr 172.21.15.6:0/2563589583 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:51:10.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:10.517+0000 7fd6a77fe700 1 -- 172.21.15.6:0/2563589583 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd698005040 con 0x7fd6a80fbad0 2024-02-09T21:51:10.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:10.517+0000 7fd6a77fe700 1 --2- 172.21.15.6:0/2563589583 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6a80fbad0 0x7fd6a810cec0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd698009b80 tx=0x7fd698009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:51:10.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:10.518+0000 7fd68ffff700 1 -- 172.21.15.6:0/2563589583 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd69800b5c0 con 0x7fd6a80fbad0 2024-02-09T21:51:10.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:10.518+0000 7fd68ffff700 1 -- 172.21.15.6:0/2563589583 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd69800be70 con 0x7fd6a80fbad0 2024-02-09T21:51:10.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:10.518+0000 7fd68ffff700 1 -- 172.21.15.6:0/2563589583 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd69801c660 con 0x7fd6a80fbad0 2024-02-09T21:51:10.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:10.518+0000 7fd6addfe700 1 -- 172.21.15.6:0/2563589583 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6a810d600 con 0x7fd6a80fbad0 2024-02-09T21:51:10.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:10.518+0000 7fd6addfe700 1 -- 172.21.15.6:0/2563589583 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a810e2a0 con 0x7fd6a80fbad0 2024-02-09T21:51:10.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:10.518+0000 7fd68ffff700 1 -- 172.21.15.6:0/2563589583 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7fd69801c8d0 con 0x7fd6a80fbad0 2024-02-09T21:51:10.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:10.518+0000 7fd6addfe700 1 -- 172.21.15.6:0/2563589583 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6a805f5c0 con 0x7fd6a80fbad0 2024-02-09T21:51:10.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:10.519+0000 7fd68ffff700 1 --2- 172.21.15.6:0/2563589583 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fd690036870 0x7fd690038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:10.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:10.519+0000 7fd68ffff700 1 -- 172.21.15.6:0/2563589583 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd698049b20 con 0x7fd6a80fbad0 2024-02-09T21:51:10.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:10.519+0000 7fd6a6ffd700 1 --2- 172.21.15.6:0/2563589583 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fd690036870 0x7fd690038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:51:10.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:10.519+0000 7fd6a6ffd700 1 --2- 172.21.15.6:0/2563589583 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fd690036870 0x7fd690038d20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd69c009990 tx=0x7fd69c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:51:10.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:10.523+0000 7fd68ffff700 1 -- 172.21.15.6:0/2563589583 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd698011550 con 0x7fd6a80fbad0 2024-02-09T21:51:10.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:10.727+0000 7fd6addfe700 1 -- 172.21.15.6:0/2563589583 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fd6a810edf0 con 0x7fd6a80fbad0 2024-02-09T21:51:11.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:11.405+0000 7fd68ffff700 1 -- 172.21.15.6:0/2563589583 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fd698011370 con 0x7fd6a80fbad0 2024-02-09T21:51:11.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:11.408+0000 7fd6addfe700 1 -- 172.21.15.6:0/2563589583 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fd690036870 msgr2=0x7fd690038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:11.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:11.408+0000 7fd6addfe700 1 --2- 172.21.15.6:0/2563589583 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fd690036870 0x7fd690038d20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd69c009990 tx=0x7fd69c006e40 comp rx=0 tx=0).stop 2024-02-09T21:51:11.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:11.408+0000 7fd6addfe700 1 -- 172.21.15.6:0/2563589583 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6a80fbad0 msgr2=0x7fd6a810cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:11.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:11.408+0000 7fd6addfe700 1 --2- 172.21.15.6:0/2563589583 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6a80fbad0 0x7fd6a810cec0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd698009b80 tx=0x7fd698009e30 comp rx=0 tx=0).stop 2024-02-09T21:51:11.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:11.408+0000 7fd6addfe700 1 -- 172.21.15.6:0/2563589583 shutdown_connections 2024-02-09T21:51:11.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:11.408+0000 7fd6addfe700 1 --2- 172.21.15.6:0/2563589583 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fd690036870 0x7fd690038d20 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:11.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:11.408+0000 7fd6addfe700 1 --2- 172.21.15.6:0/2563589583 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6a80fbad0 0x7fd6a810cec0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:11.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:11.408+0000 7fd6addfe700 1 -- 172.21.15.6:0/2563589583 >> 172.21.15.6:0/2563589583 conn(0x7fd6a80f7190 msgr2=0x7fd6a8065960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:11.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:11.408+0000 7fd6addfe700 1 -- 172.21.15.6:0/2563589583 shutdown_connections 2024-02-09T21:51:11.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:51:11.408+0000 7fd6addfe700 1 -- 172.21.15.6:0/2563589583 wait complete. 2024-02-09T21:51:11.412 INFO:teuthology.orchestra.run.smithi006.stderr:adjusted tunables profile to default 2024-02-09T21:51:11.734 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:11 smithi006 ceph-mon[29258]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:51:11.734 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:11 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/2563589583' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-09T21:51:11.735 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:11 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:51:11.735 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:11 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:51:11.735 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:11 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:51:11.735 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:11 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:51:12.017 INFO:tasks.cephadm:Adding mon.smithi006 on smithi006 2024-02-09T21:51:12.017 INFO:tasks.cephadm:Adding mon.smithi090 on smithi090 2024-02-09T21:51:12.017 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph orch apply mon '2;smithi006:172.21.15.6=smithi006;smithi090:172.21.15.90=smithi090' 2024-02-09T21:51:12.264 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-09T21:51:12.312 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-09T21:51:12.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:12 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:51:12.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:12 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/2563589583' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-09T21:51:12.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:12 smithi006 ceph-mon[29258]: osdmap e4: 0 total, 0 up, 0 in 2024-02-09T21:51:12.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:12 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:51:12.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:12 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-09T21:51:12.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:12 smithi006 ceph-mon[29258]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-09T21:51:12.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:12 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:51:12.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:12 smithi006 ceph-mon[29258]: Deploying daemon grafana.smithi006 on smithi006 2024-02-09T21:51:13.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.710+0000 7f1114279700 1 -- 172.21.15.90:0/541869600 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f110c0fe200 msgr2=0x7f110c0fe610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:13.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.710+0000 7f1114279700 1 --2- 172.21.15.90:0/541869600 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f110c0fe200 0x7f110c0fe610 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f11080048c0 tx=0x7f1108009e30 comp rx=0 tx=0).stop 2024-02-09T21:51:13.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.710+0000 7f1114279700 1 -- 172.21.15.90:0/541869600 shutdown_connections 2024-02-09T21:51:13.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.710+0000 7f1114279700 1 --2- 172.21.15.90:0/541869600 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f110c0fe200 0x7f110c0fe610 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:13.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.710+0000 7f1114279700 1 -- 172.21.15.90:0/541869600 >> 172.21.15.90:0/541869600 conn(0x7f110c0f69c0 msgr2=0x7f110c0f8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:13.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.710+0000 7f1114279700 1 -- 172.21.15.90:0/541869600 shutdown_connections 2024-02-09T21:51:13.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.710+0000 7f1114279700 1 -- 172.21.15.90:0/541869600 wait complete. 2024-02-09T21:51:13.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.710+0000 7f1114279700 1 Processor -- start 2024-02-09T21:51:13.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.710+0000 7f1114279700 1 -- start start 2024-02-09T21:51:13.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.711+0000 7f1114279700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f110c0fe200 0x7f110c10cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:13.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.711+0000 7f1114279700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f110c10d480 con 0x7f110c0fe200 2024-02-09T21:51:13.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.711+0000 7f1112015700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f110c0fe200 0x7f110c10cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:51:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.711+0000 7f1112015700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f110c0fe200 0x7f110c10cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.90:34732/0 (socket says 172.21.15.90:34732) 2024-02-09T21:51:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.711+0000 7f1112015700 1 -- 172.21.15.90:0/3068491738 learned_addr learned my addr 172.21.15.90:0/3068491738 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-09T21:51:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.711+0000 7f1112015700 1 -- 172.21.15.90:0/3068491738 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1108005040 con 0x7f110c0fe200 2024-02-09T21:51:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.711+0000 7f1112015700 1 --2- 172.21.15.90:0/3068491738 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f110c0fe200 0x7f110c10cf40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1108009300 tx=0x7f1108009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:51:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.711+0000 7f1102ffd700 1 -- 172.21.15.90:0/3068491738 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1108004e60 con 0x7f110c0fe200 2024-02-09T21:51:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.711+0000 7f1102ffd700 1 -- 172.21.15.90:0/3068491738 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1108009680 con 0x7f110c0fe200 2024-02-09T21:51:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.711+0000 7f1114279700 1 -- 172.21.15.90:0/3068491738 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f110c10d680 con 0x7f110c0fe200 2024-02-09T21:51:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.711+0000 7f1102ffd700 1 -- 172.21.15.90:0/3068491738 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f110801c5b0 con 0x7f110c0fe200 2024-02-09T21:51:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.711+0000 7f1114279700 1 -- 172.21.15.90:0/3068491738 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f110c10e320 con 0x7f110c0fe200 2024-02-09T21:51:13.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.712+0000 7f1102ffd700 1 -- 172.21.15.90:0/3068491738 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7f110801c870 con 0x7f110c0fe200 2024-02-09T21:51:13.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.712+0000 7f1114279700 1 -- 172.21.15.90:0/3068491738 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f110c05f5c0 con 0x7f110c0fe200 2024-02-09T21:51:13.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.712+0000 7f1102ffd700 1 --2- 172.21.15.90:0/3068491738 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f10f8036870 0x7f10f8038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:13.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.712+0000 7f1102ffd700 1 -- 172.21.15.90:0/3068491738 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1108049ae0 con 0x7f110c0fe200 2024-02-09T21:51:13.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.712+0000 7f1111814700 1 --2- 172.21.15.90:0/3068491738 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f10f8036870 0x7f10f8038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:51:13.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.713+0000 7f1111814700 1 --2- 172.21.15.90:0/3068491738 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f10f8036870 0x7f10f8038d20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f10fc009990 tx=0x7f10fc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:51:13.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.717+0000 7f1102ffd700 1 -- 172.21.15.90:0/3068491738 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1108023300 con 0x7f110c0fe200 2024-02-09T21:51:13.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:13 smithi006 ceph-mon[29258]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:51:13.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.906+0000 7f1114279700 1 -- 172.21.15.90:0/3068491738 --> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi006:172.21.15.6=smithi006;smithi090:172.21.15.90=smithi090", "target": ["mon-mgr", ""]}) v1 -- 0x7f110c10ef90 con 0x7f10f8036870 2024-02-09T21:51:13.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.912+0000 7f1102ffd700 1 -- 172.21.15.90:0/3068491738 <== mgr.14162 v2:172.21.15.6:6800/3321564517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f110c10ef90 con 0x7f10f8036870 2024-02-09T21:51:13.916 INFO:teuthology.orchestra.run.smithi090.stdout:Scheduled mon update... 2024-02-09T21:51:13.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.914+0000 7f1114279700 1 -- 172.21.15.90:0/3068491738 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f10f8036870 msgr2=0x7f10f8038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:13.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.914+0000 7f1114279700 1 --2- 172.21.15.90:0/3068491738 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f10f8036870 0x7f10f8038d20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f10fc009990 tx=0x7f10fc006e40 comp rx=0 tx=0).stop 2024-02-09T21:51:13.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.914+0000 7f1114279700 1 -- 172.21.15.90:0/3068491738 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f110c0fe200 msgr2=0x7f110c10cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:13.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.914+0000 7f1114279700 1 --2- 172.21.15.90:0/3068491738 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f110c0fe200 0x7f110c10cf40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1108009300 tx=0x7f1108009e30 comp rx=0 tx=0).stop 2024-02-09T21:51:13.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.914+0000 7f1114279700 1 -- 172.21.15.90:0/3068491738 shutdown_connections 2024-02-09T21:51:13.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.914+0000 7f1114279700 1 --2- 172.21.15.90:0/3068491738 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f10f8036870 0x7f10f8038d20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:13.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.914+0000 7f1114279700 1 --2- 172.21.15.90:0/3068491738 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f110c0fe200 0x7f110c10cf40 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:13.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.914+0000 7f1114279700 1 -- 172.21.15.90:0/3068491738 >> 172.21.15.90:0/3068491738 conn(0x7f110c0f69c0 msgr2=0x7f110c0f7760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:13.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.914+0000 7f1114279700 1 -- 172.21.15.90:0/3068491738 shutdown_connections 2024-02-09T21:51:13.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:13.914+0000 7f1114279700 1 -- 172.21.15.90:0/3068491738 wait complete. 2024-02-09T21:51:14.431 DEBUG:teuthology.orchestra.run.smithi090:mon.smithi090> sudo journalctl -f -n 0 -u ceph-ee26d820-c794-11ee-95b8-87774f69a715@mon.smithi090.service 2024-02-09T21:51:14.433 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-09T21:51:14.433 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-09T21:51:14.709 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-09T21:51:14.755 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-09T21:51:15.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:14 smithi006 ceph-mon[29258]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi006:172.21.15.6=smithi006;smithi090:172.21.15.90=smithi090", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:51:15.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:14 smithi006 ceph-mon[29258]: Saving service mon spec with placement smithi006:172.21.15.6=smithi006;smithi090:172.21.15.90=smithi090;count:2 2024-02-09T21:51:15.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:14 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:51:15.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:14 smithi006 ceph-mon[29258]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:51:16.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.075+0000 7f7bcb5d5700 1 -- 172.21.15.90:0/461441267 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bc40fb260 msgr2=0x7f7bc40fb670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:16.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.075+0000 7f7bcb5d5700 1 --2- 172.21.15.90:0/461441267 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bc40fb260 0x7f7bc40fb670 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7bb40049d0 tx=0x7f7bb400af40 comp rx=0 tx=0).stop 2024-02-09T21:51:16.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.076+0000 7f7bcb5d5700 1 -- 172.21.15.90:0/461441267 shutdown_connections 2024-02-09T21:51:16.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.076+0000 7f7bcb5d5700 1 --2- 172.21.15.90:0/461441267 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bc40fb260 0x7f7bc40fb670 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:16.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.076+0000 7f7bcb5d5700 1 -- 172.21.15.90:0/461441267 >> 172.21.15.90:0/461441267 conn(0x7f7bc40f6920 msgr2=0x7f7bc40f8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:16.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.076+0000 7f7bcb5d5700 1 -- 172.21.15.90:0/461441267 shutdown_connections 2024-02-09T21:51:16.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.076+0000 7f7bcb5d5700 1 -- 172.21.15.90:0/461441267 wait complete. 2024-02-09T21:51:16.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.076+0000 7f7bcb5d5700 1 Processor -- start 2024-02-09T21:51:16.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.076+0000 7f7bcb5d5700 1 -- start start 2024-02-09T21:51:16.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.076+0000 7f7bcb5d5700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bc40fb260 0x7f7bc410cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:16.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.076+0000 7f7bcb5d5700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc410d510 con 0x7f7bc40fb260 2024-02-09T21:51:16.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.076+0000 7f7bc9371700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bc40fb260 0x7f7bc410cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:51:16.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.076+0000 7f7bc9371700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bc40fb260 0x7f7bc410cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.90:34748/0 (socket says 172.21.15.90:34748) 2024-02-09T21:51:16.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.076+0000 7f7bc9371700 1 -- 172.21.15.90:0/913048746 learned_addr learned my addr 172.21.15.90:0/913048746 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-09T21:51:16.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.077+0000 7f7bc9371700 1 -- 172.21.15.90:0/913048746 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb4005040 con 0x7f7bc40fb260 2024-02-09T21:51:16.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.077+0000 7f7bc9371700 1 --2- 172.21.15.90:0/913048746 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bc40fb260 0x7f7bc410cfd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7bb400cf00 tx=0x7f7bb400af40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:51:16.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.077+0000 7f7bb9ffb700 1 -- 172.21.15.90:0/913048746 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bb400a8b0 con 0x7f7bc40fb260 2024-02-09T21:51:16.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.077+0000 7f7bb9ffb700 1 -- 172.21.15.90:0/913048746 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7bb400aa10 con 0x7f7bc40fb260 2024-02-09T21:51:16.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.077+0000 7f7bcb5d5700 1 -- 172.21.15.90:0/913048746 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bc410d710 con 0x7f7bc40fb260 2024-02-09T21:51:16.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.077+0000 7f7bb9ffb700 1 -- 172.21.15.90:0/913048746 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bb40147a0 con 0x7f7bc40fb260 2024-02-09T21:51:16.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.077+0000 7f7bcb5d5700 1 -- 172.21.15.90:0/913048746 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bc410e3b0 con 0x7f7bc40fb260 2024-02-09T21:51:16.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.078+0000 7f7bb9ffb700 1 -- 172.21.15.90:0/913048746 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7f7bb4014a20 con 0x7f7bc40fb260 2024-02-09T21:51:16.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.078+0000 7f7bb9ffb700 1 --2- 172.21.15.90:0/913048746 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f7bb00368c0 0x7f7bb0038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:16.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.078+0000 7f7bb9ffb700 1 -- 172.21.15.90:0/913048746 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f7bb404a250 con 0x7f7bc40fb260 2024-02-09T21:51:16.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.078+0000 7f7bcb5d5700 1 -- 172.21.15.90:0/913048746 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bc405f5c0 con 0x7f7bc40fb260 2024-02-09T21:51:16.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.078+0000 7f7bc8b70700 1 --2- 172.21.15.90:0/913048746 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f7bb00368c0 0x7f7bb0038d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:51:16.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.079+0000 7f7bc8b70700 1 --2- 172.21.15.90:0/913048746 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f7bb00368c0 0x7f7bb0038d70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7bc0009990 tx=0x7f7bc0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:51:16.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.083+0000 7f7bb9ffb700 1 -- 172.21.15.90:0/913048746 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7bb401c3e0 con 0x7f7bc40fb260 2024-02-09T21:51:16.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.339+0000 7f7bcb5d5700 1 -- 172.21.15.90:0/913048746 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7bc410ee80 con 0x7f7bc40fb260 2024-02-09T21:51:16.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.340+0000 7f7bb9ffb700 1 -- 172.21.15.90:0/913048746 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f7bb4012370 con 0x7f7bc40fb260 2024-02-09T21:51:16.343 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:51:16.343 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"ee26d820-c794-11ee-95b8-87774f69a715","modified":"2024-02-09T21:49:36.342734Z","created":"2024-02-09T21:49:36.342734Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi006","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3300","nonce":0},{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-09T21:51:16.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.342+0000 7f7bcb5d5700 1 -- 172.21.15.90:0/913048746 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f7bb00368c0 msgr2=0x7f7bb0038d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:16.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.342+0000 7f7bcb5d5700 1 --2- 172.21.15.90:0/913048746 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f7bb00368c0 0x7f7bb0038d70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7bc0009990 tx=0x7f7bc0006e40 comp rx=0 tx=0).stop 2024-02-09T21:51:16.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.342+0000 7f7bcb5d5700 1 -- 172.21.15.90:0/913048746 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bc40fb260 msgr2=0x7f7bc410cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:16.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.342+0000 7f7bcb5d5700 1 --2- 172.21.15.90:0/913048746 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bc40fb260 0x7f7bc410cfd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7bb400cf00 tx=0x7f7bb400af40 comp rx=0 tx=0).stop 2024-02-09T21:51:16.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.342+0000 7f7bcb5d5700 1 -- 172.21.15.90:0/913048746 shutdown_connections 2024-02-09T21:51:16.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.342+0000 7f7bcb5d5700 1 --2- 172.21.15.90:0/913048746 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f7bb00368c0 0x7f7bb0038d70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:16.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.342+0000 7f7bcb5d5700 1 --2- 172.21.15.90:0/913048746 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bc40fb260 0x7f7bc410cfd0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:16.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.342+0000 7f7bcb5d5700 1 -- 172.21.15.90:0/913048746 >> 172.21.15.90:0/913048746 conn(0x7f7bc40f6920 msgr2=0x7f7bc40f7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:16.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.342+0000 7f7bcb5d5700 1 -- 172.21.15.90:0/913048746 shutdown_connections 2024-02-09T21:51:16.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:16.342+0000 7f7bcb5d5700 1 -- 172.21.15.90:0/913048746 wait complete. 2024-02-09T21:51:16.346 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-02-09T21:51:16.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:16 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:51:17.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:17 smithi006 ceph-mon[29258]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:51:17.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:17 smithi006 ceph-mon[29258]: from='client.? 172.21.15.90:0/913048746' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T21:51:17.889 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-09T21:51:17.890 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-09T21:51:18.137 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-09T21:51:18.182 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-09T21:51:19.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.626+0000 7f741c444700 1 -- 172.21.15.90:0/1148412086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74140fbac0 msgr2=0x7f74140fbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:19.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.626+0000 7f741c444700 1 --2- 172.21.15.90:0/1148412086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74140fbac0 0x7f74140fbed0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f74100048c0 tx=0x7f7410009e30 comp rx=0 tx=0).stop 2024-02-09T21:51:19.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.626+0000 7f741c444700 1 -- 172.21.15.90:0/1148412086 shutdown_connections 2024-02-09T21:51:19.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.626+0000 7f741c444700 1 --2- 172.21.15.90:0/1148412086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74140fbac0 0x7f74140fbed0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:19.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.626+0000 7f741c444700 1 -- 172.21.15.90:0/1148412086 >> 172.21.15.90:0/1148412086 conn(0x7f74140f71a0 msgr2=0x7f74140f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:19.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.626+0000 7f741c444700 1 -- 172.21.15.90:0/1148412086 shutdown_connections 2024-02-09T21:51:19.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.626+0000 7f741c444700 1 -- 172.21.15.90:0/1148412086 wait complete. 2024-02-09T21:51:19.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.626+0000 7f741c444700 1 Processor -- start 2024-02-09T21:51:19.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.626+0000 7f741c444700 1 -- start start 2024-02-09T21:51:19.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.627+0000 7f741c444700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74140fbac0 0x7f741410fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:19.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.627+0000 7f741c444700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7414066850 con 0x7f74140fbac0 2024-02-09T21:51:19.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.627+0000 7f741a1e0700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74140fbac0 0x7f741410fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:51:19.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.627+0000 7f741a1e0700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74140fbac0 0x7f741410fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.90:53534/0 (socket says 172.21.15.90:53534) 2024-02-09T21:51:19.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.627+0000 7f741a1e0700 1 -- 172.21.15.90:0/2025166560 learned_addr learned my addr 172.21.15.90:0/2025166560 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-09T21:51:19.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.627+0000 7f741a1e0700 1 -- 172.21.15.90:0/2025166560 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7410005040 con 0x7f74140fbac0 2024-02-09T21:51:19.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.627+0000 7f741a1e0700 1 --2- 172.21.15.90:0/2025166560 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74140fbac0 0x7f741410fb90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7410009680 tx=0x7f7410009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:51:19.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.627+0000 7f740affd700 1 -- 172.21.15.90:0/2025166560 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f741000bed0 con 0x7f74140fbac0 2024-02-09T21:51:19.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.627+0000 7f740affd700 1 -- 172.21.15.90:0/2025166560 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f741000b5c0 con 0x7f74140fbac0 2024-02-09T21:51:19.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.627+0000 7f741c444700 1 -- 172.21.15.90:0/2025166560 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7414066a50 con 0x7f74140fbac0 2024-02-09T21:51:19.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.627+0000 7f740affd700 1 -- 172.21.15.90:0/2025166560 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f741001c5b0 con 0x7f74140fbac0 2024-02-09T21:51:19.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.627+0000 7f741c444700 1 -- 172.21.15.90:0/2025166560 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7414110310 con 0x7f74140fbac0 2024-02-09T21:51:19.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.628+0000 7f740affd700 1 -- 172.21.15.90:0/2025166560 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7f741001c820 con 0x7f74140fbac0 2024-02-09T21:51:19.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.628+0000 7f740affd700 1 --2- 172.21.15.90:0/2025166560 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f7400036870 0x7f7400038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:19.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.628+0000 7f741c444700 1 -- 172.21.15.90:0/2025166560 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7414105890 con 0x7f74140fbac0 2024-02-09T21:51:19.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.628+0000 7f740affd700 1 -- 172.21.15.90:0/2025166560 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f7410049b70 con 0x7f74140fbac0 2024-02-09T21:51:19.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.628+0000 7f74199df700 1 --2- 172.21.15.90:0/2025166560 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f7400036870 0x7f7400038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:51:19.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.629+0000 7f74199df700 1 --2- 172.21.15.90:0/2025166560 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f7400036870 0x7f7400038d20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7404009990 tx=0x7f7404006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:51:19.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.633+0000 7f740affd700 1 -- 172.21.15.90:0/2025166560 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7410023300 con 0x7f74140fbac0 2024-02-09T21:51:19.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:19 smithi006 ceph-mon[29258]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:51:19.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.891+0000 7f741c444700 1 -- 172.21.15.90:0/2025166560 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f741402b3e0 con 0x7f74140fbac0 2024-02-09T21:51:19.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.891+0000 7f740affd700 1 -- 172.21.15.90:0/2025166560 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f7410034090 con 0x7f74140fbac0 2024-02-09T21:51:19.895 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:51:19.895 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"ee26d820-c794-11ee-95b8-87774f69a715","modified":"2024-02-09T21:49:36.342734Z","created":"2024-02-09T21:49:36.342734Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi006","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3300","nonce":0},{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-09T21:51:19.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.893+0000 7f741c444700 1 -- 172.21.15.90:0/2025166560 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f7400036870 msgr2=0x7f7400038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:19.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.893+0000 7f741c444700 1 --2- 172.21.15.90:0/2025166560 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f7400036870 0x7f7400038d20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7404009990 tx=0x7f7404006e40 comp rx=0 tx=0).stop 2024-02-09T21:51:19.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.893+0000 7f741c444700 1 -- 172.21.15.90:0/2025166560 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74140fbac0 msgr2=0x7f741410fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:19.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.893+0000 7f741c444700 1 --2- 172.21.15.90:0/2025166560 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74140fbac0 0x7f741410fb90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7410009680 tx=0x7f7410009e30 comp rx=0 tx=0).stop 2024-02-09T21:51:19.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.893+0000 7f741c444700 1 -- 172.21.15.90:0/2025166560 shutdown_connections 2024-02-09T21:51:19.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.893+0000 7f741c444700 1 --2- 172.21.15.90:0/2025166560 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f7400036870 0x7f7400038d20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:19.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.893+0000 7f741c444700 1 --2- 172.21.15.90:0/2025166560 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74140fbac0 0x7f741410fb90 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:19.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.893+0000 7f741c444700 1 -- 172.21.15.90:0/2025166560 >> 172.21.15.90:0/2025166560 conn(0x7f74140f71a0 msgr2=0x7f74140f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:19.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.893+0000 7f741c444700 1 -- 172.21.15.90:0/2025166560 shutdown_connections 2024-02-09T21:51:19.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:19.893+0000 7f741c444700 1 -- 172.21.15.90:0/2025166560 wait complete. 2024-02-09T21:51:19.898 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-02-09T21:51:20.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:20 smithi006 ceph-mon[29258]: from='client.? 172.21.15.90:0/2025166560' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T21:51:21.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:21 smithi006 ceph-mon[29258]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:51:21.466 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-09T21:51:21.466 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-09T21:51:21.714 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-09T21:51:21.761 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-09T21:51:23.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.235+0000 7ff7f18f6700 1 -- 172.21.15.90:0/4172465603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7ec0fcbb0 msgr2=0x7ff7ec0fcfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:23.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.235+0000 7ff7f18f6700 1 --2- 172.21.15.90:0/4172465603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7ec0fcbb0 0x7ff7ec0fcfc0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff7e00048c0 tx=0x7ff7e0009e30 comp rx=0 tx=0).stop 2024-02-09T21:51:23.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.235+0000 7ff7f18f6700 1 -- 172.21.15.90:0/4172465603 shutdown_connections 2024-02-09T21:51:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.235+0000 7ff7f18f6700 1 --2- 172.21.15.90:0/4172465603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7ec0fcbb0 0x7ff7ec0fcfc0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.235+0000 7ff7f18f6700 1 -- 172.21.15.90:0/4172465603 >> 172.21.15.90:0/4172465603 conn(0x7ff7ec0f8330 msgr2=0x7ff7ec0fa740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.235+0000 7ff7f18f6700 1 -- 172.21.15.90:0/4172465603 shutdown_connections 2024-02-09T21:51:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.235+0000 7ff7f18f6700 1 -- 172.21.15.90:0/4172465603 wait complete. 2024-02-09T21:51:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.236+0000 7ff7f18f6700 1 Processor -- start 2024-02-09T21:51:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.236+0000 7ff7f18f6700 1 -- start start 2024-02-09T21:51:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.236+0000 7ff7f18f6700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7ec0fcbb0 0x7ff7ec10ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.236+0000 7ff7f18f6700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ec10d3d0 con 0x7ff7ec0fcbb0 2024-02-09T21:51:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.236+0000 7ff7eaffd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7ec0fcbb0 0x7ff7ec10ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:51:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.236+0000 7ff7eaffd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7ec0fcbb0 0x7ff7ec10ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.90:53546/0 (socket says 172.21.15.90:53546) 2024-02-09T21:51:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.236+0000 7ff7eaffd700 1 -- 172.21.15.90:0/2079684951 learned_addr learned my addr 172.21.15.90:0/2079684951 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-09T21:51:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.237+0000 7ff7eaffd700 1 -- 172.21.15.90:0/2079684951 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e0005040 con 0x7ff7ec0fcbb0 2024-02-09T21:51:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.237+0000 7ff7eaffd700 1 --2- 172.21.15.90:0/2079684951 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7ec0fcbb0 0x7ff7ec10ce90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff7e0009680 tx=0x7ff7e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:51:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.237+0000 7ff7d37fe700 1 -- 172.21.15.90:0/2079684951 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7e000b5c0 con 0x7ff7ec0fcbb0 2024-02-09T21:51:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.237+0000 7ff7f18f6700 1 -- 172.21.15.90:0/2079684951 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7ec10d5d0 con 0x7ff7ec0fcbb0 2024-02-09T21:51:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.237+0000 7ff7d37fe700 1 -- 172.21.15.90:0/2079684951 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff7e0004e60 con 0x7ff7ec0fcbb0 2024-02-09T21:51:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.237+0000 7ff7d37fe700 1 -- 172.21.15.90:0/2079684951 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7e001c590 con 0x7ff7ec0fcbb0 2024-02-09T21:51:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.237+0000 7ff7f18f6700 1 -- 172.21.15.90:0/2079684951 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7ec10e270 con 0x7ff7ec0fcbb0 2024-02-09T21:51:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.238+0000 7ff7d37fe700 1 -- 172.21.15.90:0/2079684951 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7ff7e0013420 con 0x7ff7ec0fcbb0 2024-02-09T21:51:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.238+0000 7ff7f18f6700 1 -- 172.21.15.90:0/2079684951 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7ec05f5c0 con 0x7ff7ec0fcbb0 2024-02-09T21:51:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.238+0000 7ff7d37fe700 1 --2- 172.21.15.90:0/2079684951 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7ff7d4036870 0x7ff7d4038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:23.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.238+0000 7ff7d37fe700 1 -- 172.21.15.90:0/2079684951 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ff7e0049bb0 con 0x7ff7ec0fcbb0 2024-02-09T21:51:23.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.238+0000 7ff7ea7fc700 1 --2- 172.21.15.90:0/2079684951 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7ff7d4036870 0x7ff7d4038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:51:23.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.239+0000 7ff7ea7fc700 1 --2- 172.21.15.90:0/2079684951 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7ff7d4036870 0x7ff7d4038d20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc009990 tx=0x7ff7dc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:51:23.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.243+0000 7ff7d37fe700 1 -- 172.21.15.90:0/2079684951 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff7e0025400 con 0x7ff7ec0fcbb0 2024-02-09T21:51:23.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.496+0000 7ff7f18f6700 1 -- 172.21.15.90:0/2079684951 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff7ec10edc0 con 0x7ff7ec0fcbb0 2024-02-09T21:51:23.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.497+0000 7ff7d37fe700 1 -- 172.21.15.90:0/2079684951 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7ff7e0011370 con 0x7ff7ec0fcbb0 2024-02-09T21:51:23.501 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:51:23.501 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"ee26d820-c794-11ee-95b8-87774f69a715","modified":"2024-02-09T21:49:36.342734Z","created":"2024-02-09T21:49:36.342734Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi006","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3300","nonce":0},{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-09T21:51:23.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.499+0000 7ff7f18f6700 1 -- 172.21.15.90:0/2079684951 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7ff7d4036870 msgr2=0x7ff7d4038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:23.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.499+0000 7ff7f18f6700 1 --2- 172.21.15.90:0/2079684951 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7ff7d4036870 0x7ff7d4038d20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc009990 tx=0x7ff7dc006e40 comp rx=0 tx=0).stop 2024-02-09T21:51:23.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.499+0000 7ff7f18f6700 1 -- 172.21.15.90:0/2079684951 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7ec0fcbb0 msgr2=0x7ff7ec10ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:23.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.499+0000 7ff7f18f6700 1 --2- 172.21.15.90:0/2079684951 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7ec0fcbb0 0x7ff7ec10ce90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff7e0009680 tx=0x7ff7e0009e30 comp rx=0 tx=0).stop 2024-02-09T21:51:23.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.499+0000 7ff7f18f6700 1 -- 172.21.15.90:0/2079684951 shutdown_connections 2024-02-09T21:51:23.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.499+0000 7ff7f18f6700 1 --2- 172.21.15.90:0/2079684951 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7ff7d4036870 0x7ff7d4038d20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:23.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.499+0000 7ff7f18f6700 1 --2- 172.21.15.90:0/2079684951 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7ec0fcbb0 0x7ff7ec10ce90 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:23.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.499+0000 7ff7f18f6700 1 -- 172.21.15.90:0/2079684951 >> 172.21.15.90:0/2079684951 conn(0x7ff7ec0f8330 msgr2=0x7ff7ec0f9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:23.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.499+0000 7ff7f18f6700 1 -- 172.21.15.90:0/2079684951 shutdown_connections 2024-02-09T21:51:23.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:23.499+0000 7ff7f18f6700 1 -- 172.21.15.90:0/2079684951 wait complete. 2024-02-09T21:51:23.503 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-02-09T21:51:23.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:23 smithi006 ceph-mon[29258]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:51:24.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:24 smithi006 ceph-mon[29258]: from='client.? 172.21.15.90:0/2079684951' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T21:51:25.234 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-09T21:51:25.234 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-09T21:51:25.483 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-09T21:51:25.529 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-09T21:51:25.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:25 smithi006 ceph-mon[29258]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:51:26.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:26.868+0000 7fb29f962700 1 -- 172.21.15.90:0/288565096 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2980fbad0 msgr2=0x7fb2980fbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:26.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:26.868+0000 7fb29f962700 1 --2- 172.21.15.90:0/288565096 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2980fbad0 0x7fb2980fbee0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb2880048c0 tx=0x7fb288009e30 comp rx=0 tx=0).stop 2024-02-09T21:51:26.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:26.868+0000 7fb29f962700 1 -- 172.21.15.90:0/288565096 shutdown_connections 2024-02-09T21:51:26.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:26.868+0000 7fb29f962700 1 --2- 172.21.15.90:0/288565096 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2980fbad0 0x7fb2980fbee0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:26.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:26.868+0000 7fb29f962700 1 -- 172.21.15.90:0/288565096 >> 172.21.15.90:0/288565096 conn(0x7fb2980f7190 msgr2=0x7fb2980f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:26.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:26.869+0000 7fb29f962700 1 -- 172.21.15.90:0/288565096 shutdown_connections 2024-02-09T21:51:26.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:26.869+0000 7fb29f962700 1 -- 172.21.15.90:0/288565096 wait complete. 2024-02-09T21:51:26.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:26.869+0000 7fb29f962700 1 Processor -- start 2024-02-09T21:51:26.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:26.869+0000 7fb29f962700 1 -- start start 2024-02-09T21:51:26.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:26.869+0000 7fb29f962700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2980fbad0 0x7fb29810cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:26.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:26.869+0000 7fb29f962700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29810d420 con 0x7fb2980fbad0 2024-02-09T21:51:26.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:26.869+0000 7fb29d6fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2980fbad0 0x7fb29810cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:51:26.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:26.869+0000 7fb29d6fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2980fbad0 0x7fb29810cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.90:53568/0 (socket says 172.21.15.90:53568) 2024-02-09T21:51:26.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:26.869+0000 7fb29d6fe700 1 -- 172.21.15.90:0/1152925548 learned_addr learned my addr 172.21.15.90:0/1152925548 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-09T21:51:26.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:26.870+0000 7fb29d6fe700 1 -- 172.21.15.90:0/1152925548 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb288005040 con 0x7fb2980fbad0 2024-02-09T21:51:26.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:26.870+0000 7fb29d6fe700 1 --2- 172.21.15.90:0/1152925548 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2980fbad0 0x7fb29810cee0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb288009b80 tx=0x7fb288009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:51:26.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:26.870+0000 7fb28dffb700 1 -- 172.21.15.90:0/1152925548 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb28800b5c0 con 0x7fb2980fbad0 2024-02-09T21:51:26.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:26.870+0000 7fb29f962700 1 -- 172.21.15.90:0/1152925548 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb29810d620 con 0x7fb2980fbad0 2024-02-09T21:51:26.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:26.870+0000 7fb28dffb700 1 -- 172.21.15.90:0/1152925548 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb28800be70 con 0x7fb2980fbad0 2024-02-09T21:51:26.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:26.870+0000 7fb28dffb700 1 -- 172.21.15.90:0/1152925548 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb28801c660 con 0x7fb2980fbad0 2024-02-09T21:51:26.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:26.870+0000 7fb29f962700 1 -- 172.21.15.90:0/1152925548 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb29810e2c0 con 0x7fb2980fbad0 2024-02-09T21:51:26.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:26.871+0000 7fb28dffb700 1 -- 172.21.15.90:0/1152925548 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7fb28801c8d0 con 0x7fb2980fbad0 2024-02-09T21:51:26.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:26.871+0000 7fb28dffb700 1 --2- 172.21.15.90:0/1152925548 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fb284036870 0x7fb284038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:26.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:26.871+0000 7fb29f962700 1 -- 172.21.15.90:0/1152925548 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb29805f5c0 con 0x7fb2980fbad0 2024-02-09T21:51:26.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:26.871+0000 7fb28dffb700 1 -- 172.21.15.90:0/1152925548 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb288049ad0 con 0x7fb2980fbad0 2024-02-09T21:51:26.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:26.871+0000 7fb29cefd700 1 --2- 172.21.15.90:0/1152925548 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fb284036870 0x7fb284038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:51:26.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:26.872+0000 7fb29cefd700 1 --2- 172.21.15.90:0/1152925548 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fb284036870 0x7fb284038d20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb294009990 tx=0x7fb294006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:51:26.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:26.878+0000 7fb28dffb700 1 -- 172.21.15.90:0/1152925548 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb288011dd0 con 0x7fb2980fbad0 2024-02-09T21:51:27.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:27.134+0000 7fb29f962700 1 -- 172.21.15.90:0/1152925548 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb29810ede0 con 0x7fb2980fbad0 2024-02-09T21:51:27.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:27.135+0000 7fb28dffb700 1 -- 172.21.15.90:0/1152925548 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7fb288011370 con 0x7fb2980fbad0 2024-02-09T21:51:27.138 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:51:27.139 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"ee26d820-c794-11ee-95b8-87774f69a715","modified":"2024-02-09T21:49:36.342734Z","created":"2024-02-09T21:49:36.342734Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi006","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3300","nonce":0},{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-09T21:51:27.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:27.137+0000 7fb29f962700 1 -- 172.21.15.90:0/1152925548 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fb284036870 msgr2=0x7fb284038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:27.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:27.137+0000 7fb29f962700 1 --2- 172.21.15.90:0/1152925548 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fb284036870 0x7fb284038d20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb294009990 tx=0x7fb294006e40 comp rx=0 tx=0).stop 2024-02-09T21:51:27.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:27.137+0000 7fb29f962700 1 -- 172.21.15.90:0/1152925548 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2980fbad0 msgr2=0x7fb29810cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:27.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:27.137+0000 7fb29f962700 1 --2- 172.21.15.90:0/1152925548 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2980fbad0 0x7fb29810cee0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb288009b80 tx=0x7fb288009e30 comp rx=0 tx=0).stop 2024-02-09T21:51:27.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:27.137+0000 7fb29f962700 1 -- 172.21.15.90:0/1152925548 shutdown_connections 2024-02-09T21:51:27.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:27.137+0000 7fb29f962700 1 --2- 172.21.15.90:0/1152925548 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fb284036870 0x7fb284038d20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:27.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:27.137+0000 7fb29f962700 1 --2- 172.21.15.90:0/1152925548 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2980fbad0 0x7fb29810cee0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:27.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:27.137+0000 7fb29f962700 1 -- 172.21.15.90:0/1152925548 >> 172.21.15.90:0/1152925548 conn(0x7fb2980f7190 msgr2=0x7fb298066490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:27.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:27.137+0000 7fb29f962700 1 -- 172.21.15.90:0/1152925548 shutdown_connections 2024-02-09T21:51:27.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:27.137+0000 7fb29f962700 1 -- 172.21.15.90:0/1152925548 wait complete. 2024-02-09T21:51:27.141 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-02-09T21:51:27.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:27 smithi006 ceph-mon[29258]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:51:27.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:27 smithi006 ceph-mon[29258]: from='client.? 172.21.15.90:0/1152925548' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T21:51:28.763 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-09T21:51:28.764 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-09T21:51:29.012 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-09T21:51:29.059 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-09T21:51:29.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:29 smithi006 ceph-mon[29258]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:51:30.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.437+0000 7fa5bc67b700 1 -- 172.21.15.90:0/304340518 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5b40fae80 msgr2=0x7fa5b40fb290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:30.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.437+0000 7fa5bc67b700 1 --2- 172.21.15.90:0/304340518 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5b40fae80 0x7fa5b40fb290 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa5b00048c0 tx=0x7fa5b0009e30 comp rx=0 tx=0).stop 2024-02-09T21:51:30.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.437+0000 7fa5bc67b700 1 -- 172.21.15.90:0/304340518 shutdown_connections 2024-02-09T21:51:30.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.437+0000 7fa5bc67b700 1 --2- 172.21.15.90:0/304340518 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5b40fae80 0x7fa5b40fb290 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:30.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.437+0000 7fa5bc67b700 1 -- 172.21.15.90:0/304340518 >> 172.21.15.90:0/304340518 conn(0x7fa5b40f6920 msgr2=0x7fa5b40f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:30.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.437+0000 7fa5bc67b700 1 -- 172.21.15.90:0/304340518 shutdown_connections 2024-02-09T21:51:30.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.437+0000 7fa5bc67b700 1 -- 172.21.15.90:0/304340518 wait complete. 2024-02-09T21:51:30.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.437+0000 7fa5bc67b700 1 Processor -- start 2024-02-09T21:51:30.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.437+0000 7fa5bc67b700 1 -- start start 2024-02-09T21:51:30.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.437+0000 7fa5bc67b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5b40fae80 0x7fa5b40fd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:30.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.437+0000 7fa5bc67b700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b40fbe20 con 0x7fa5b40fae80 2024-02-09T21:51:30.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.438+0000 7fa5ba417700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5b40fae80 0x7fa5b40fd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:51:30.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.438+0000 7fa5ba417700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5b40fae80 0x7fa5b40fd9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.90:49486/0 (socket says 172.21.15.90:49486) 2024-02-09T21:51:30.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.438+0000 7fa5ba417700 1 -- 172.21.15.90:0/3541592815 learned_addr learned my addr 172.21.15.90:0/3541592815 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-09T21:51:30.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.438+0000 7fa5ba417700 1 -- 172.21.15.90:0/3541592815 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b0005040 con 0x7fa5b40fae80 2024-02-09T21:51:30.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.438+0000 7fa5ba417700 1 --2- 172.21.15.90:0/3541592815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5b40fae80 0x7fa5b40fd9a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0009040 tx=0x7fa5b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:51:30.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.438+0000 7fa5aaffd700 1 -- 172.21.15.90:0/3541592815 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5b000bed0 con 0x7fa5b40fae80 2024-02-09T21:51:30.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.438+0000 7fa5bc67b700 1 -- 172.21.15.90:0/3541592815 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5b40fc020 con 0x7fa5b40fae80 2024-02-09T21:51:30.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.438+0000 7fa5aaffd700 1 -- 172.21.15.90:0/3541592815 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa5b000b5c0 con 0x7fa5b40fae80 2024-02-09T21:51:30.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.438+0000 7fa5aaffd700 1 -- 172.21.15.90:0/3541592815 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5b001c5b0 con 0x7fa5b40fae80 2024-02-09T21:51:30.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.438+0000 7fa5bc67b700 1 -- 172.21.15.90:0/3541592815 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5b40fc510 con 0x7fa5b40fae80 2024-02-09T21:51:30.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.439+0000 7fa5aaffd700 1 -- 172.21.15.90:0/3541592815 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7fa5b0004d30 con 0x7fa5b40fae80 2024-02-09T21:51:30.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.439+0000 7fa5bc67b700 1 -- 172.21.15.90:0/3541592815 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5b4106930 con 0x7fa5b40fae80 2024-02-09T21:51:30.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.439+0000 7fa5aaffd700 1 --2- 172.21.15.90:0/3541592815 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fa5a0036870 0x7fa5a0038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:30.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.439+0000 7fa5aaffd700 1 -- 172.21.15.90:0/3541592815 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa5b0049aa0 con 0x7fa5b40fae80 2024-02-09T21:51:30.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.440+0000 7fa5b9c16700 1 --2- 172.21.15.90:0/3541592815 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fa5a0036870 0x7fa5a0038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:51:30.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.440+0000 7fa5b9c16700 1 --2- 172.21.15.90:0/3541592815 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fa5a0036870 0x7fa5a0038d20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4009990 tx=0x7fa5a4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:51:30.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.444+0000 7fa5aaffd700 1 -- 172.21.15.90:0/3541592815 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa5b0011550 con 0x7fa5b40fae80 2024-02-09T21:51:30.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.696+0000 7fa5bc67b700 1 -- 172.21.15.90:0/3541592815 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa5b404bed0 con 0x7fa5b40fae80 2024-02-09T21:51:30.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.697+0000 7fa5aaffd700 1 -- 172.21.15.90:0/3541592815 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7fa5b0011370 con 0x7fa5b40fae80 2024-02-09T21:51:30.701 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:51:30.701 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"ee26d820-c794-11ee-95b8-87774f69a715","modified":"2024-02-09T21:49:36.342734Z","created":"2024-02-09T21:49:36.342734Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi006","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3300","nonce":0},{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-09T21:51:30.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.700+0000 7fa5bc67b700 1 -- 172.21.15.90:0/3541592815 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fa5a0036870 msgr2=0x7fa5a0038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:30.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.700+0000 7fa5bc67b700 1 --2- 172.21.15.90:0/3541592815 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fa5a0036870 0x7fa5a0038d20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4009990 tx=0x7fa5a4006e40 comp rx=0 tx=0).stop 2024-02-09T21:51:30.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.700+0000 7fa5bc67b700 1 -- 172.21.15.90:0/3541592815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5b40fae80 msgr2=0x7fa5b40fd9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:30.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.700+0000 7fa5bc67b700 1 --2- 172.21.15.90:0/3541592815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5b40fae80 0x7fa5b40fd9a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0009040 tx=0x7fa5b0009e30 comp rx=0 tx=0).stop 2024-02-09T21:51:30.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.700+0000 7fa5bc67b700 1 -- 172.21.15.90:0/3541592815 shutdown_connections 2024-02-09T21:51:30.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.700+0000 7fa5bc67b700 1 --2- 172.21.15.90:0/3541592815 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fa5a0036870 0x7fa5a0038d20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:30.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.700+0000 7fa5bc67b700 1 --2- 172.21.15.90:0/3541592815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5b40fae80 0x7fa5b40fd9a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:30.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.700+0000 7fa5bc67b700 1 -- 172.21.15.90:0/3541592815 >> 172.21.15.90:0/3541592815 conn(0x7fa5b40f6920 msgr2=0x7fa5b40f76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:30.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.700+0000 7fa5bc67b700 1 -- 172.21.15.90:0/3541592815 shutdown_connections 2024-02-09T21:51:30.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:30.700+0000 7fa5bc67b700 1 -- 172.21.15.90:0/3541592815 wait complete. 2024-02-09T21:51:30.705 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-02-09T21:51:31.510 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:31 smithi006 ceph-mon[29258]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:51:31.510 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:31 smithi006 ceph-mon[29258]: from='client.? 172.21.15.90:0/3541592815' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T21:51:32.257 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-09T21:51:32.258 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-09T21:51:32.509 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-09T21:51:32.555 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-09T21:51:33.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:32 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:51:33.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:32 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:51:33.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:32 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:51:33.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:32 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:51:33.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:32 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:51:33.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:32 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:51:33.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:32 smithi006 ceph-mon[29258]: Deploying daemon prometheus.smithi006 on smithi006 2024-02-09T21:51:33.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:32 smithi006 ceph-mon[29258]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:51:33.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:33.878+0000 7f4133371700 1 -- 172.21.15.90:0/1212373416 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f412c0fe200 msgr2=0x7f412c0fe610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:33.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:33.878+0000 7f4133371700 1 --2- 172.21.15.90:0/1212373416 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f412c0fe200 0x7f412c0fe610 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f41240048c0 tx=0x7f4124009e30 comp rx=0 tx=0).stop 2024-02-09T21:51:33.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:33.878+0000 7f4133371700 1 -- 172.21.15.90:0/1212373416 shutdown_connections 2024-02-09T21:51:33.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:33.878+0000 7f4133371700 1 --2- 172.21.15.90:0/1212373416 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f412c0fe200 0x7f412c0fe610 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:33.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:33.878+0000 7f4133371700 1 -- 172.21.15.90:0/1212373416 >> 172.21.15.90:0/1212373416 conn(0x7f412c0f69c0 msgr2=0x7f412c0f8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:33.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:33.878+0000 7f4133371700 1 -- 172.21.15.90:0/1212373416 shutdown_connections 2024-02-09T21:51:33.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:33.878+0000 7f4133371700 1 -- 172.21.15.90:0/1212373416 wait complete. 2024-02-09T21:51:33.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:33.879+0000 7f4133371700 1 Processor -- start 2024-02-09T21:51:33.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:33.879+0000 7f4133371700 1 -- start start 2024-02-09T21:51:33.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:33.879+0000 7f4133371700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f412c0fe200 0x7f412c10cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:33.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:33.879+0000 7f4133371700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412c10d460 con 0x7f412c0fe200 2024-02-09T21:51:33.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:33.879+0000 7f413110d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f412c0fe200 0x7f412c10cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:51:33.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:33.879+0000 7f413110d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f412c0fe200 0x7f412c10cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.90:49506/0 (socket says 172.21.15.90:49506) 2024-02-09T21:51:33.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:33.879+0000 7f413110d700 1 -- 172.21.15.90:0/2052936142 learned_addr learned my addr 172.21.15.90:0/2052936142 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-09T21:51:33.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:33.879+0000 7f413110d700 1 -- 172.21.15.90:0/2052936142 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4124005040 con 0x7f412c0fe200 2024-02-09T21:51:33.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:33.880+0000 7f413110d700 1 --2- 172.21.15.90:0/2052936142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f412c0fe200 0x7f412c10cf20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4124009300 tx=0x7f4124009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:51:33.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:33.880+0000 7f4129ffb700 1 -- 172.21.15.90:0/2052936142 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4124004e60 con 0x7f412c0fe200 2024-02-09T21:51:33.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:33.880+0000 7f4129ffb700 1 -- 172.21.15.90:0/2052936142 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4124009680 con 0x7f412c0fe200 2024-02-09T21:51:33.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:33.880+0000 7f4133371700 1 -- 172.21.15.90:0/2052936142 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f412c10d660 con 0x7f412c0fe200 2024-02-09T21:51:33.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:33.880+0000 7f4129ffb700 1 -- 172.21.15.90:0/2052936142 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f412401c5b0 con 0x7f412c0fe200 2024-02-09T21:51:33.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:33.880+0000 7f4133371700 1 -- 172.21.15.90:0/2052936142 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f412c10e300 con 0x7f412c0fe200 2024-02-09T21:51:33.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:33.881+0000 7f4129ffb700 1 -- 172.21.15.90:0/2052936142 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7f412401c870 con 0x7f412c0fe200 2024-02-09T21:51:33.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:33.881+0000 7f4133371700 1 -- 172.21.15.90:0/2052936142 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f412c05f5c0 con 0x7f412c0fe200 2024-02-09T21:51:33.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:33.881+0000 7f4129ffb700 1 --2- 172.21.15.90:0/2052936142 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f4114036870 0x7f4114038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:33.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:33.881+0000 7f4129ffb700 1 -- 172.21.15.90:0/2052936142 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4124049b80 con 0x7f412c0fe200 2024-02-09T21:51:33.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:33.881+0000 7f413090c700 1 --2- 172.21.15.90:0/2052936142 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f4114036870 0x7f4114038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:51:33.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:33.882+0000 7f413090c700 1 --2- 172.21.15.90:0/2052936142 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f4114036870 0x7f4114038d20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4120009990 tx=0x7f4120006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:51:33.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:33.887+0000 7f4129ffb700 1 -- 172.21.15.90:0/2052936142 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4124013420 con 0x7f412c0fe200 2024-02-09T21:51:34.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:34.144+0000 7f4133371700 1 -- 172.21.15.90:0/2052936142 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f412c10ee50 con 0x7f412c0fe200 2024-02-09T21:51:34.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:34.145+0000 7f4129ffb700 1 -- 172.21.15.90:0/2052936142 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f4124011370 con 0x7f412c0fe200 2024-02-09T21:51:34.148 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:51:34.148 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"ee26d820-c794-11ee-95b8-87774f69a715","modified":"2024-02-09T21:49:36.342734Z","created":"2024-02-09T21:49:36.342734Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi006","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3300","nonce":0},{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-09T21:51:34.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:34.146+0000 7f4133371700 1 -- 172.21.15.90:0/2052936142 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f4114036870 msgr2=0x7f4114038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:34.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:34.146+0000 7f4133371700 1 --2- 172.21.15.90:0/2052936142 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f4114036870 0x7f4114038d20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4120009990 tx=0x7f4120006e40 comp rx=0 tx=0).stop 2024-02-09T21:51:34.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:34.146+0000 7f4133371700 1 -- 172.21.15.90:0/2052936142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f412c0fe200 msgr2=0x7f412c10cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:34.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:34.146+0000 7f4133371700 1 --2- 172.21.15.90:0/2052936142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f412c0fe200 0x7f412c10cf20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4124009300 tx=0x7f4124009e30 comp rx=0 tx=0).stop 2024-02-09T21:51:34.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:34.147+0000 7f4133371700 1 -- 172.21.15.90:0/2052936142 shutdown_connections 2024-02-09T21:51:34.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:34.147+0000 7f4133371700 1 --2- 172.21.15.90:0/2052936142 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f4114036870 0x7f4114038d20 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:34.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:34.147+0000 7f4133371700 1 --2- 172.21.15.90:0/2052936142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f412c0fe200 0x7f412c10cf20 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:34.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:34.147+0000 7f4133371700 1 -- 172.21.15.90:0/2052936142 >> 172.21.15.90:0/2052936142 conn(0x7f412c0f69c0 msgr2=0x7f412c0f7760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:34.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:34.147+0000 7f4133371700 1 -- 172.21.15.90:0/2052936142 shutdown_connections 2024-02-09T21:51:34.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:34.147+0000 7f4133371700 1 -- 172.21.15.90:0/2052936142 wait complete. 2024-02-09T21:51:34.151 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-02-09T21:51:34.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:34 smithi006 ceph-mon[29258]: from='client.? 172.21.15.90:0/2052936142' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T21:51:35.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:35 smithi006 ceph-mon[29258]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:51:35.739 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-09T21:51:35.739 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-09T21:51:35.991 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-09T21:51:36.038 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-09T21:51:36.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:36 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:51:37.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.342+0000 7f5b1b08f700 1 -- 172.21.15.90:0/2486572580 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b140fcbd0 msgr2=0x7f5b140fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:37.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.342+0000 7f5b1b08f700 1 --2- 172.21.15.90:0/2486572580 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b140fcbd0 0x7f5b140fcfe0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5b080048c0 tx=0x7f5b08009e30 comp rx=0 tx=0).stop 2024-02-09T21:51:37.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.342+0000 7f5b1b08f700 1 -- 172.21.15.90:0/2486572580 shutdown_connections 2024-02-09T21:51:37.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.342+0000 7f5b1b08f700 1 --2- 172.21.15.90:0/2486572580 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b140fcbd0 0x7f5b140fcfe0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:37.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.342+0000 7f5b1b08f700 1 -- 172.21.15.90:0/2486572580 >> 172.21.15.90:0/2486572580 conn(0x7f5b140f8290 msgr2=0x7f5b140fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:37.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.342+0000 7f5b1b08f700 1 -- 172.21.15.90:0/2486572580 shutdown_connections 2024-02-09T21:51:37.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.342+0000 7f5b1b08f700 1 -- 172.21.15.90:0/2486572580 wait complete. 2024-02-09T21:51:37.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.343+0000 7f5b1b08f700 1 Processor -- start 2024-02-09T21:51:37.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.343+0000 7f5b1b08f700 1 -- start start 2024-02-09T21:51:37.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.343+0000 7f5b1b08f700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b140fcbd0 0x7f5b1410cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:37.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.343+0000 7f5b1b08f700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1410d400 con 0x7f5b140fcbd0 2024-02-09T21:51:37.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.343+0000 7f5b18e2b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b140fcbd0 0x7f5b1410cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:51:37.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.343+0000 7f5b18e2b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b140fcbd0 0x7f5b1410cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.90:49520/0 (socket says 172.21.15.90:49520) 2024-02-09T21:51:37.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.343+0000 7f5b18e2b700 1 -- 172.21.15.90:0/1597250604 learned_addr learned my addr 172.21.15.90:0/1597250604 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-09T21:51:37.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.343+0000 7f5b18e2b700 1 -- 172.21.15.90:0/1597250604 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b08005040 con 0x7f5b140fcbd0 2024-02-09T21:51:37.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.344+0000 7f5b18e2b700 1 --2- 172.21.15.90:0/1597250604 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b140fcbd0 0x7f5b1410cec0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5b08009b80 tx=0x7f5b08009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:51:37.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.344+0000 7f5b117fa700 1 -- 172.21.15.90:0/1597250604 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b0800b5c0 con 0x7f5b140fcbd0 2024-02-09T21:51:37.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.344+0000 7f5b1b08f700 1 -- 172.21.15.90:0/1597250604 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b1410d600 con 0x7f5b140fcbd0 2024-02-09T21:51:37.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.344+0000 7f5b117fa700 1 -- 172.21.15.90:0/1597250604 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5b0800be70 con 0x7f5b140fcbd0 2024-02-09T21:51:37.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.344+0000 7f5b117fa700 1 -- 172.21.15.90:0/1597250604 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b0801c660 con 0x7f5b140fcbd0 2024-02-09T21:51:37.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.344+0000 7f5b1b08f700 1 -- 172.21.15.90:0/1597250604 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b1410e2a0 con 0x7f5b140fcbd0 2024-02-09T21:51:37.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.345+0000 7f5b117fa700 1 -- 172.21.15.90:0/1597250604 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7f5b0801c8d0 con 0x7f5b140fcbd0 2024-02-09T21:51:37.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.345+0000 7f5b117fa700 1 --2- 172.21.15.90:0/1597250604 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f5b04036870 0x7f5b04038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:37.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.345+0000 7f5b1b08f700 1 -- 172.21.15.90:0/1597250604 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b1405f5c0 con 0x7f5b140fcbd0 2024-02-09T21:51:37.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.345+0000 7f5b117fa700 1 -- 172.21.15.90:0/1597250604 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5b08049ba0 con 0x7f5b140fcbd0 2024-02-09T21:51:37.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.345+0000 7f5b13fff700 1 --2- 172.21.15.90:0/1597250604 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f5b04036870 0x7f5b04038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:51:37.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.345+0000 7f5b13fff700 1 --2- 172.21.15.90:0/1597250604 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f5b04036870 0x7f5b04038d20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5b00009990 tx=0x7f5b00006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:51:37.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.350+0000 7f5b117fa700 1 -- 172.21.15.90:0/1597250604 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f5b080118d0 con 0x7f5b140fcbd0 2024-02-09T21:51:37.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.605+0000 7f5b1b08f700 1 -- 172.21.15.90:0/1597250604 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5b1410edf0 con 0x7f5b140fcbd0 2024-02-09T21:51:37.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.606+0000 7f5b117fa700 1 -- 172.21.15.90:0/1597250604 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f5b08004d30 con 0x7f5b140fcbd0 2024-02-09T21:51:37.609 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:51:37.610 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"ee26d820-c794-11ee-95b8-87774f69a715","modified":"2024-02-09T21:49:36.342734Z","created":"2024-02-09T21:49:36.342734Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi006","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3300","nonce":0},{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-09T21:51:37.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.608+0000 7f5b1b08f700 1 -- 172.21.15.90:0/1597250604 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f5b04036870 msgr2=0x7f5b04038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:37.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.608+0000 7f5b1b08f700 1 --2- 172.21.15.90:0/1597250604 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f5b04036870 0x7f5b04038d20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5b00009990 tx=0x7f5b00006e40 comp rx=0 tx=0).stop 2024-02-09T21:51:37.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.608+0000 7f5b1b08f700 1 -- 172.21.15.90:0/1597250604 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b140fcbd0 msgr2=0x7f5b1410cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:37.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.608+0000 7f5b1b08f700 1 --2- 172.21.15.90:0/1597250604 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b140fcbd0 0x7f5b1410cec0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5b08009b80 tx=0x7f5b08009e30 comp rx=0 tx=0).stop 2024-02-09T21:51:37.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.608+0000 7f5b1b08f700 1 -- 172.21.15.90:0/1597250604 shutdown_connections 2024-02-09T21:51:37.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.608+0000 7f5b1b08f700 1 --2- 172.21.15.90:0/1597250604 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f5b04036870 0x7f5b04038d20 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:37.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.608+0000 7f5b1b08f700 1 --2- 172.21.15.90:0/1597250604 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b140fcbd0 0x7f5b1410cec0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:37.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.608+0000 7f5b1b08f700 1 -- 172.21.15.90:0/1597250604 >> 172.21.15.90:0/1597250604 conn(0x7f5b140f8290 msgr2=0x7f5b140f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:37.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.608+0000 7f5b1b08f700 1 -- 172.21.15.90:0/1597250604 shutdown_connections 2024-02-09T21:51:37.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:37.608+0000 7f5b1b08f700 1 -- 172.21.15.90:0/1597250604 wait complete. 2024-02-09T21:51:37.612 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-02-09T21:51:37.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:37 smithi006 ceph-mon[29258]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:51:38.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:38 smithi006 ceph-mon[29258]: from='client.? 172.21.15.90:0/1597250604' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T21:51:39.215 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-09T21:51:39.215 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-09T21:51:39.463 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-09T21:51:39.508 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-09T21:51:39.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:39 smithi006 ceph-mon[29258]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:51:40.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:40.944+0000 7efc32b1a700 1 -- 172.21.15.90:0/3697330221 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc2c0fcbd0 msgr2=0x7efc2c0fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:40.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:40.944+0000 7efc32b1a700 1 --2- 172.21.15.90:0/3697330221 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc2c0fcbd0 0x7efc2c0fcfe0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efc24009c60 tx=0x7efc24004d10 comp rx=0 tx=0).stop 2024-02-09T21:51:40.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:40.944+0000 7efc32b1a700 1 -- 172.21.15.90:0/3697330221 shutdown_connections 2024-02-09T21:51:40.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:40.944+0000 7efc32b1a700 1 --2- 172.21.15.90:0/3697330221 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc2c0fcbd0 0x7efc2c0fcfe0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:40.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:40.944+0000 7efc32b1a700 1 -- 172.21.15.90:0/3697330221 >> 172.21.15.90:0/3697330221 conn(0x7efc2c0f8270 msgr2=0x7efc2c0fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:40.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:40.944+0000 7efc32b1a700 1 -- 172.21.15.90:0/3697330221 shutdown_connections 2024-02-09T21:51:40.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:40.944+0000 7efc32b1a700 1 -- 172.21.15.90:0/3697330221 wait complete. 2024-02-09T21:51:40.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:40.945+0000 7efc32b1a700 1 Processor -- start 2024-02-09T21:51:40.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:40.945+0000 7efc32b1a700 1 -- start start 2024-02-09T21:51:40.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:40.945+0000 7efc32b1a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc2c0fcbd0 0x7efc2c10cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:40.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:40.945+0000 7efc32b1a700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc2c10d4a0 con 0x7efc2c0fcbd0 2024-02-09T21:51:40.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:40.945+0000 7efc308b6700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc2c0fcbd0 0x7efc2c10cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:51:40.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:40.945+0000 7efc308b6700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc2c0fcbd0 0x7efc2c10cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.90:38712/0 (socket says 172.21.15.90:38712) 2024-02-09T21:51:40.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:40.945+0000 7efc308b6700 1 -- 172.21.15.90:0/1263750888 learned_addr learned my addr 172.21.15.90:0/1263750888 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-09T21:51:40.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:40.946+0000 7efc308b6700 1 -- 172.21.15.90:0/1263750888 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc24009940 con 0x7efc2c0fcbd0 2024-02-09T21:51:40.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:40.946+0000 7efc308b6700 1 --2- 172.21.15.90:0/1263750888 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc2c0fcbd0 0x7efc2c10cf60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efc24015040 tx=0x7efc24004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:51:40.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:40.946+0000 7efc297fa700 1 -- 172.21.15.90:0/1263750888 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc24007c80 con 0x7efc2c0fcbd0 2024-02-09T21:51:40.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:40.946+0000 7efc32b1a700 1 -- 172.21.15.90:0/1263750888 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc2c10d6a0 con 0x7efc2c0fcbd0 2024-02-09T21:51:40.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:40.946+0000 7efc297fa700 1 -- 172.21.15.90:0/1263750888 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efc2401d910 con 0x7efc2c0fcbd0 2024-02-09T21:51:40.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:40.946+0000 7efc297fa700 1 -- 172.21.15.90:0/1263750888 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc240129f0 con 0x7efc2c0fcbd0 2024-02-09T21:51:40.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:40.946+0000 7efc32b1a700 1 -- 172.21.15.90:0/1263750888 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc2c10e2f0 con 0x7efc2c0fcbd0 2024-02-09T21:51:40.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:40.947+0000 7efc297fa700 1 -- 172.21.15.90:0/1263750888 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7efc24019040 con 0x7efc2c0fcbd0 2024-02-09T21:51:40.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:40.947+0000 7efc32b1a700 1 -- 172.21.15.90:0/1263750888 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc2c05f5c0 con 0x7efc2c0fcbd0 2024-02-09T21:51:40.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:40.947+0000 7efc297fa700 1 --2- 172.21.15.90:0/1263750888 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7efc18036870 0x7efc18038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:40.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:40.947+0000 7efc297fa700 1 -- 172.21.15.90:0/1263750888 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7efc2404a3d0 con 0x7efc2c0fcbd0 2024-02-09T21:51:40.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:40.947+0000 7efc2bfff700 1 --2- 172.21.15.90:0/1263750888 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7efc18036870 0x7efc18038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:51:40.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:40.948+0000 7efc2bfff700 1 --2- 172.21.15.90:0/1263750888 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7efc18036870 0x7efc18038d20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7efc1c009990 tx=0x7efc1c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:51:40.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:40.952+0000 7efc297fa700 1 -- 172.21.15.90:0/1263750888 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7efc2402eb40 con 0x7efc2c0fcbd0 2024-02-09T21:51:41.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:41.206+0000 7efc32b1a700 1 -- 172.21.15.90:0/1263750888 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efc2c10ede0 con 0x7efc2c0fcbd0 2024-02-09T21:51:41.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:41.207+0000 7efc297fa700 1 -- 172.21.15.90:0/1263750888 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7efc24017070 con 0x7efc2c0fcbd0 2024-02-09T21:51:41.210 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:51:41.211 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"ee26d820-c794-11ee-95b8-87774f69a715","modified":"2024-02-09T21:49:36.342734Z","created":"2024-02-09T21:49:36.342734Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi006","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3300","nonce":0},{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-09T21:51:41.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:41.209+0000 7efc32b1a700 1 -- 172.21.15.90:0/1263750888 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7efc18036870 msgr2=0x7efc18038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:41.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:41.209+0000 7efc32b1a700 1 --2- 172.21.15.90:0/1263750888 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7efc18036870 0x7efc18038d20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7efc1c009990 tx=0x7efc1c006e40 comp rx=0 tx=0).stop 2024-02-09T21:51:41.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:41.209+0000 7efc32b1a700 1 -- 172.21.15.90:0/1263750888 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc2c0fcbd0 msgr2=0x7efc2c10cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:41.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:41.209+0000 7efc32b1a700 1 --2- 172.21.15.90:0/1263750888 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc2c0fcbd0 0x7efc2c10cf60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efc24015040 tx=0x7efc24004d10 comp rx=0 tx=0).stop 2024-02-09T21:51:41.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:41.209+0000 7efc32b1a700 1 -- 172.21.15.90:0/1263750888 shutdown_connections 2024-02-09T21:51:41.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:41.209+0000 7efc32b1a700 1 --2- 172.21.15.90:0/1263750888 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7efc18036870 0x7efc18038d20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:41.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:41.209+0000 7efc32b1a700 1 --2- 172.21.15.90:0/1263750888 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc2c0fcbd0 0x7efc2c10cf60 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:41.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:41.209+0000 7efc32b1a700 1 -- 172.21.15.90:0/1263750888 >> 172.21.15.90:0/1263750888 conn(0x7efc2c0f8270 msgr2=0x7efc2c0f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:41.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:41.209+0000 7efc32b1a700 1 -- 172.21.15.90:0/1263750888 shutdown_connections 2024-02-09T21:51:41.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:41.209+0000 7efc32b1a700 1 -- 172.21.15.90:0/1263750888 wait complete. 2024-02-09T21:51:41.213 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-02-09T21:51:41.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:41 smithi006 ceph-mon[29258]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:51:41.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:41 smithi006 ceph-mon[29258]: from='client.? 172.21.15.90:0/1263750888' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T21:51:42.781 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-09T21:51:42.781 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-09T21:51:43.028 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-09T21:51:43.075 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-09T21:51:43.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:43 smithi006 ceph-mon[29258]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:51:44.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.337+0000 7f1598111700 1 -- 172.21.15.90:0/2182871180 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15900fbab0 msgr2=0x7f15900fbec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:44.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.337+0000 7f1598111700 1 --2- 172.21.15.90:0/2182871180 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15900fbab0 0x7f15900fbec0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f15800048c0 tx=0x7f1580009e30 comp rx=0 tx=0).stop 2024-02-09T21:51:44.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.337+0000 7f1598111700 1 -- 172.21.15.90:0/2182871180 shutdown_connections 2024-02-09T21:51:44.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.337+0000 7f1598111700 1 --2- 172.21.15.90:0/2182871180 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15900fbab0 0x7f15900fbec0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:44.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.337+0000 7f1598111700 1 -- 172.21.15.90:0/2182871180 >> 172.21.15.90:0/2182871180 conn(0x7f15900f7210 msgr2=0x7f15900f9640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:44.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.337+0000 7f1598111700 1 -- 172.21.15.90:0/2182871180 shutdown_connections 2024-02-09T21:51:44.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.337+0000 7f1598111700 1 -- 172.21.15.90:0/2182871180 wait complete. 2024-02-09T21:51:44.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.338+0000 7f1598111700 1 Processor -- start 2024-02-09T21:51:44.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.338+0000 7f1598111700 1 -- start start 2024-02-09T21:51:44.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.338+0000 7f1598111700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15900fbab0 0x7f159010be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:44.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.338+0000 7f1598111700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f159010c370 con 0x7f15900fbab0 2024-02-09T21:51:44.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.338+0000 7f1595ead700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15900fbab0 0x7f159010be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:51:44.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.338+0000 7f1595ead700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15900fbab0 0x7f159010be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.90:38734/0 (socket says 172.21.15.90:38734) 2024-02-09T21:51:44.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.338+0000 7f1595ead700 1 -- 172.21.15.90:0/1493956842 learned_addr learned my addr 172.21.15.90:0/1493956842 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-09T21:51:44.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.338+0000 7f1595ead700 1 -- 172.21.15.90:0/1493956842 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1580005040 con 0x7f15900fbab0 2024-02-09T21:51:44.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.339+0000 7f1595ead700 1 --2- 172.21.15.90:0/1493956842 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15900fbab0 0x7f159010be30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1580009b80 tx=0x7f1580009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:51:44.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.339+0000 7f15867fc700 1 -- 172.21.15.90:0/1493956842 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f158000b5c0 con 0x7f15900fbab0 2024-02-09T21:51:44.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.339+0000 7f1598111700 1 -- 172.21.15.90:0/1493956842 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f159010c570 con 0x7f15900fbab0 2024-02-09T21:51:44.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.339+0000 7f15867fc700 1 -- 172.21.15.90:0/1493956842 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f158000be70 con 0x7f15900fbab0 2024-02-09T21:51:44.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.339+0000 7f15867fc700 1 -- 172.21.15.90:0/1493956842 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f158001c660 con 0x7f15900fbab0 2024-02-09T21:51:44.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.339+0000 7f1598111700 1 -- 172.21.15.90:0/1493956842 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f159010d130 con 0x7f15900fbab0 2024-02-09T21:51:44.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.340+0000 7f15867fc700 1 -- 172.21.15.90:0/1493956842 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7f1580004d30 con 0x7f15900fbab0 2024-02-09T21:51:44.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.340+0000 7f1598111700 1 -- 172.21.15.90:0/1493956842 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f159005f5c0 con 0x7f15900fbab0 2024-02-09T21:51:44.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.340+0000 7f15867fc700 1 --2- 172.21.15.90:0/1493956842 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f157c0368c0 0x7f157c038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:44.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.340+0000 7f15867fc700 1 -- 172.21.15.90:0/1493956842 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1580049a60 con 0x7f15900fbab0 2024-02-09T21:51:44.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.340+0000 7f15956ac700 1 --2- 172.21.15.90:0/1493956842 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f157c0368c0 0x7f157c038d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:51:44.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.341+0000 7f15956ac700 1 --2- 172.21.15.90:0/1493956842 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f157c0368c0 0x7f157c038d70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f158c009990 tx=0x7f158c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:51:44.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.345+0000 7f15867fc700 1 -- 172.21.15.90:0/1493956842 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1580025400 con 0x7f15900fbab0 2024-02-09T21:51:44.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.599+0000 7f1598111700 1 -- 172.21.15.90:0/1493956842 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f159010dd00 con 0x7f15900fbab0 2024-02-09T21:51:44.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.599+0000 7f15867fc700 1 -- 172.21.15.90:0/1493956842 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f1580011370 con 0x7f15900fbab0 2024-02-09T21:51:44.603 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:51:44.603 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"ee26d820-c794-11ee-95b8-87774f69a715","modified":"2024-02-09T21:49:36.342734Z","created":"2024-02-09T21:49:36.342734Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi006","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3300","nonce":0},{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-09T21:51:44.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.601+0000 7f1598111700 1 -- 172.21.15.90:0/1493956842 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f157c0368c0 msgr2=0x7f157c038d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:44.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.601+0000 7f1598111700 1 --2- 172.21.15.90:0/1493956842 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f157c0368c0 0x7f157c038d70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f158c009990 tx=0x7f158c006e40 comp rx=0 tx=0).stop 2024-02-09T21:51:44.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.601+0000 7f1598111700 1 -- 172.21.15.90:0/1493956842 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15900fbab0 msgr2=0x7f159010be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:44.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.601+0000 7f1598111700 1 --2- 172.21.15.90:0/1493956842 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15900fbab0 0x7f159010be30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1580009b80 tx=0x7f1580009e30 comp rx=0 tx=0).stop 2024-02-09T21:51:44.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.601+0000 7f1598111700 1 -- 172.21.15.90:0/1493956842 shutdown_connections 2024-02-09T21:51:44.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.601+0000 7f1598111700 1 --2- 172.21.15.90:0/1493956842 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7f157c0368c0 0x7f157c038d70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:44.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.601+0000 7f1598111700 1 --2- 172.21.15.90:0/1493956842 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15900fbab0 0x7f159010be30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:44.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.601+0000 7f1598111700 1 -- 172.21.15.90:0/1493956842 >> 172.21.15.90:0/1493956842 conn(0x7f15900f7210 msgr2=0x7f15900f7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:44.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.601+0000 7f1598111700 1 -- 172.21.15.90:0/1493956842 shutdown_connections 2024-02-09T21:51:44.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:44.601+0000 7f1598111700 1 -- 172.21.15.90:0/1493956842 wait complete. 2024-02-09T21:51:44.606 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-02-09T21:51:45.612 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:45 smithi006 ceph-mon[29258]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:51:45.612 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:45 smithi006 ceph-mon[29258]: from='client.? 172.21.15.90:0/1493956842' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T21:51:46.164 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-09T21:51:46.164 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-09T21:51:46.412 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-09T21:51:46.458 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-09T21:51:47.480 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:47 smithi006 ceph-mon[29258]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:51:47.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:47.861+0000 7fbf2d8a2700 1 -- 172.21.15.90:0/2894796627 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf280fbab0 msgr2=0x7fbf280fbec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:47.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:47.861+0000 7fbf2d8a2700 1 --2- 172.21.15.90:0/2894796627 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf280fbab0 0x7fbf280fbec0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbf180048c0 tx=0x7fbf18009e30 comp rx=0 tx=0).stop 2024-02-09T21:51:47.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:47.861+0000 7fbf2d8a2700 1 -- 172.21.15.90:0/2894796627 shutdown_connections 2024-02-09T21:51:47.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:47.861+0000 7fbf2d8a2700 1 --2- 172.21.15.90:0/2894796627 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf280fbab0 0x7fbf280fbec0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:47.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:47.861+0000 7fbf2d8a2700 1 -- 172.21.15.90:0/2894796627 >> 172.21.15.90:0/2894796627 conn(0x7fbf280f7210 msgr2=0x7fbf280f9640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:47.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:47.861+0000 7fbf2d8a2700 1 -- 172.21.15.90:0/2894796627 shutdown_connections 2024-02-09T21:51:47.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:47.861+0000 7fbf2d8a2700 1 -- 172.21.15.90:0/2894796627 wait complete. 2024-02-09T21:51:47.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:47.862+0000 7fbf2d8a2700 1 Processor -- start 2024-02-09T21:51:47.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:47.862+0000 7fbf2d8a2700 1 -- start start 2024-02-09T21:51:47.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:47.862+0000 7fbf2d8a2700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf280fbab0 0x7fbf280f92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:47.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:47.862+0000 7fbf2d8a2700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf280f7750 con 0x7fbf280fbab0 2024-02-09T21:51:47.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:47.862+0000 7fbf26ffd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf280fbab0 0x7fbf280f92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:51:47.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:47.862+0000 7fbf26ffd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf280fbab0 0x7fbf280f92d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.90:57512/0 (socket says 172.21.15.90:57512) 2024-02-09T21:51:47.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:47.862+0000 7fbf26ffd700 1 -- 172.21.15.90:0/1549993311 learned_addr learned my addr 172.21.15.90:0/1549993311 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-09T21:51:47.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:47.863+0000 7fbf26ffd700 1 -- 172.21.15.90:0/1549993311 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf18005040 con 0x7fbf280fbab0 2024-02-09T21:51:47.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:47.863+0000 7fbf26ffd700 1 --2- 172.21.15.90:0/1549993311 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf280fbab0 0x7fbf280f92d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbf18009680 tx=0x7fbf18009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:51:47.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:47.863+0000 7fbf1f7fe700 1 -- 172.21.15.90:0/1549993311 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf1800b5c0 con 0x7fbf280fbab0 2024-02-09T21:51:47.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:47.863+0000 7fbf2d8a2700 1 -- 172.21.15.90:0/1549993311 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf280f7950 con 0x7fbf280fbab0 2024-02-09T21:51:47.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:47.863+0000 7fbf1f7fe700 1 -- 172.21.15.90:0/1549993311 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbf18004e60 con 0x7fbf280fbab0 2024-02-09T21:51:47.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:47.863+0000 7fbf1f7fe700 1 -- 172.21.15.90:0/1549993311 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf1801c590 con 0x7fbf280fbab0 2024-02-09T21:51:47.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:47.863+0000 7fbf2d8a2700 1 -- 172.21.15.90:0/1549993311 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf280f7e40 con 0x7fbf280fbab0 2024-02-09T21:51:47.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:47.864+0000 7fbf1f7fe700 1 -- 172.21.15.90:0/1549993311 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7fbf1801c810 con 0x7fbf280fbab0 2024-02-09T21:51:47.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:47.864+0000 7fbf2d8a2700 1 -- 172.21.15.90:0/1549993311 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf28106810 con 0x7fbf280fbab0 2024-02-09T21:51:47.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:47.864+0000 7fbf1f7fe700 1 --2- 172.21.15.90:0/1549993311 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fbf14036870 0x7fbf14038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:47.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:47.864+0000 7fbf1f7fe700 1 -- 172.21.15.90:0/1549993311 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fbf18049bc0 con 0x7fbf280fbab0 2024-02-09T21:51:47.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:47.864+0000 7fbf267fc700 1 --2- 172.21.15.90:0/1549993311 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fbf14036870 0x7fbf14038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:51:47.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:47.865+0000 7fbf267fc700 1 --2- 172.21.15.90:0/1549993311 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fbf14036870 0x7fbf14038d20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbf10009990 tx=0x7fbf10006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:51:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:47.869+0000 7fbf1f7fe700 1 -- 172.21.15.90:0/1549993311 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fbf18023300 con 0x7fbf280fbab0 2024-02-09T21:51:48.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:48.123+0000 7fbf2d8a2700 1 -- 172.21.15.90:0/1549993311 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbf2805f5c0 con 0x7fbf280fbab0 2024-02-09T21:51:48.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:48.124+0000 7fbf1f7fe700 1 -- 172.21.15.90:0/1549993311 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7fbf18011b60 con 0x7fbf280fbab0 2024-02-09T21:51:48.128 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:51:48.128 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"ee26d820-c794-11ee-95b8-87774f69a715","modified":"2024-02-09T21:49:36.342734Z","created":"2024-02-09T21:49:36.342734Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi006","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3300","nonce":0},{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-09T21:51:48.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:48.126+0000 7fbf2d8a2700 1 -- 172.21.15.90:0/1549993311 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fbf14036870 msgr2=0x7fbf14038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:48.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:48.126+0000 7fbf2d8a2700 1 --2- 172.21.15.90:0/1549993311 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fbf14036870 0x7fbf14038d20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbf10009990 tx=0x7fbf10006e40 comp rx=0 tx=0).stop 2024-02-09T21:51:48.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:48.126+0000 7fbf2d8a2700 1 -- 172.21.15.90:0/1549993311 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf280fbab0 msgr2=0x7fbf280f92d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:48.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:48.126+0000 7fbf2d8a2700 1 --2- 172.21.15.90:0/1549993311 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf280fbab0 0x7fbf280f92d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbf18009680 tx=0x7fbf18009e30 comp rx=0 tx=0).stop 2024-02-09T21:51:48.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:48.126+0000 7fbf2d8a2700 1 -- 172.21.15.90:0/1549993311 shutdown_connections 2024-02-09T21:51:48.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:48.126+0000 7fbf2d8a2700 1 --2- 172.21.15.90:0/1549993311 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fbf14036870 0x7fbf14038d20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:48.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:48.126+0000 7fbf2d8a2700 1 --2- 172.21.15.90:0/1549993311 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf280fbab0 0x7fbf280f92d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:48.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:48.126+0000 7fbf2d8a2700 1 -- 172.21.15.90:0/1549993311 >> 172.21.15.90:0/1549993311 conn(0x7fbf280f7210 msgr2=0x7fbf28066430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:48.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:48.126+0000 7fbf2d8a2700 1 -- 172.21.15.90:0/1549993311 shutdown_connections 2024-02-09T21:51:48.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:48.126+0000 7fbf2d8a2700 1 -- 172.21.15.90:0/1549993311 wait complete. 2024-02-09T21:51:48.130 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-02-09T21:51:48.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:48 smithi006 ceph-mon[29258]: from='client.? 172.21.15.90:0/1549993311' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T21:51:49.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:49 smithi006 ceph-mon[29258]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:51:49.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:49 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:51:49.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:49 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:51:49.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:49 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' 2024-02-09T21:51:49.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:49 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-09T21:51:49.681 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-09T21:51:49.681 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-09T21:51:49.928 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-09T21:51:49.974 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-09T21:51:51.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.385+0000 7fd85e368700 1 -- 172.21.15.90:0/3534347483 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8580fae80 msgr2=0x7fd8580fb290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:51.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.385+0000 7fd85e368700 1 --2- 172.21.15.90:0/3534347483 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8580fae80 0x7fd8580fb290 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd8480048c0 tx=0x7fd848009e30 comp rx=0 tx=0).stop 2024-02-09T21:51:51.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.385+0000 7fd85e368700 1 -- 172.21.15.90:0/3534347483 shutdown_connections 2024-02-09T21:51:51.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.385+0000 7fd85e368700 1 --2- 172.21.15.90:0/3534347483 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8580fae80 0x7fd8580fb290 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:51.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.385+0000 7fd85e368700 1 -- 172.21.15.90:0/3534347483 >> 172.21.15.90:0/3534347483 conn(0x7fd8580f6920 msgr2=0x7fd8580f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:51.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.385+0000 7fd85e368700 1 -- 172.21.15.90:0/3534347483 shutdown_connections 2024-02-09T21:51:51.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.385+0000 7fd85e368700 1 -- 172.21.15.90:0/3534347483 wait complete. 2024-02-09T21:51:51.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.386+0000 7fd85e368700 1 Processor -- start 2024-02-09T21:51:51.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.386+0000 7fd85e368700 1 -- start start 2024-02-09T21:51:51.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.386+0000 7fd85e368700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8580fae80 0x7fd85810ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:51.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.386+0000 7fd85e368700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85810d410 con 0x7fd8580fae80 2024-02-09T21:51:51.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.386+0000 7fd857fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8580fae80 0x7fd85810ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:51:51.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.386+0000 7fd857fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8580fae80 0x7fd85810ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.90:57526/0 (socket says 172.21.15.90:57526) 2024-02-09T21:51:51.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.386+0000 7fd857fff700 1 -- 172.21.15.90:0/3428623319 learned_addr learned my addr 172.21.15.90:0/3428623319 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-09T21:51:51.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.387+0000 7fd857fff700 1 -- 172.21.15.90:0/3428623319 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd848005040 con 0x7fd8580fae80 2024-02-09T21:51:51.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.387+0000 7fd857fff700 1 --2- 172.21.15.90:0/3428623319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8580fae80 0x7fd85810ced0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd848000c00 tx=0x7fd848009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:51:51.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.387+0000 7fd854ff9700 1 -- 172.21.15.90:0/3428623319 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd84800b5c0 con 0x7fd8580fae80 2024-02-09T21:51:51.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.387+0000 7fd854ff9700 1 -- 172.21.15.90:0/3428623319 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd84800be70 con 0x7fd8580fae80 2024-02-09T21:51:51.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.387+0000 7fd85e368700 1 -- 172.21.15.90:0/3428623319 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd85810d610 con 0x7fd8580fae80 2024-02-09T21:51:51.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.387+0000 7fd854ff9700 1 -- 172.21.15.90:0/3428623319 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd84801c590 con 0x7fd8580fae80 2024-02-09T21:51:51.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.387+0000 7fd85e368700 1 -- 172.21.15.90:0/3428623319 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd85810e1d0 con 0x7fd8580fae80 2024-02-09T21:51:51.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.388+0000 7fd854ff9700 1 -- 172.21.15.90:0/3428623319 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 14) v1 ==== 43437+0+0 (secure 0 0 0) 0x7fd848013a80 con 0x7fd8580fae80 2024-02-09T21:51:51.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.388+0000 7fd85e368700 1 -- 172.21.15.90:0/3428623319 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd85805f5c0 con 0x7fd8580fae80 2024-02-09T21:51:51.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.388+0000 7fd854ff9700 1 --2- 172.21.15.90:0/3428623319 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fd8400368c0 0x7fd840038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:51.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.388+0000 7fd854ff9700 1 -- 172.21.15.90:0/3428623319 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd848049ac0 con 0x7fd8580fae80 2024-02-09T21:51:51.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.388+0000 7fd8577fe700 1 -- 172.21.15.90:0/3428623319 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fd8400368c0 msgr2=0x7fd840038d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/3321564517 2024-02-09T21:51:51.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.388+0000 7fd8577fe700 1 --2- 172.21.15.90:0/3428623319 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fd8400368c0 0x7fd840038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-09T21:51:51.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.393+0000 7fd854ff9700 1 -- 172.21.15.90:0/3428623319 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd84804be60 con 0x7fd8580fae80 2024-02-09T21:51:51.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:51 smithi006 ceph-mon[29258]: from='mgr.14162 172.21.15.6:0/1131346221' entity='mgr.smithi006.glirxt' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-09T21:51:51.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:51 smithi006 ceph-mon[29258]: mgrmap e14: smithi006.glirxt(active, since 69s) 2024-02-09T21:51:51.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.589+0000 7fd8577fe700 1 -- 172.21.15.90:0/3428623319 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fd8400368c0 msgr2=0x7fd840038d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/3321564517 2024-02-09T21:51:51.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.589+0000 7fd8577fe700 1 --2- 172.21.15.90:0/3428623319 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fd8400368c0 0x7fd840038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-09T21:51:51.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.648+0000 7fd85e368700 1 -- 172.21.15.90:0/3428623319 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd85810ee10 con 0x7fd8580fae80 2024-02-09T21:51:51.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.649+0000 7fd854ff9700 1 -- 172.21.15.90:0/3428623319 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7fd848011550 con 0x7fd8580fae80 2024-02-09T21:51:51.652 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:51:51.652 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"ee26d820-c794-11ee-95b8-87774f69a715","modified":"2024-02-09T21:49:36.342734Z","created":"2024-02-09T21:49:36.342734Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi006","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3300","nonce":0},{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-09T21:51:51.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.650+0000 7fd85e368700 1 -- 172.21.15.90:0/3428623319 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fd8400368c0 msgr2=0x7fd840038d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-09T21:51:51.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.650+0000 7fd85e368700 1 --2- 172.21.15.90:0/3428623319 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fd8400368c0 0x7fd840038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:51.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.650+0000 7fd85e368700 1 -- 172.21.15.90:0/3428623319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8580fae80 msgr2=0x7fd85810ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:51.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.650+0000 7fd85e368700 1 --2- 172.21.15.90:0/3428623319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8580fae80 0x7fd85810ced0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd848000c00 tx=0x7fd848009e30 comp rx=0 tx=0).stop 2024-02-09T21:51:51.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.651+0000 7fd85e368700 1 -- 172.21.15.90:0/3428623319 shutdown_connections 2024-02-09T21:51:51.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.651+0000 7fd85e368700 1 --2- 172.21.15.90:0/3428623319 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7fd8400368c0 0x7fd840038d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:51.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.651+0000 7fd85e368700 1 --2- 172.21.15.90:0/3428623319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8580fae80 0x7fd85810ced0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:51.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.651+0000 7fd85e368700 1 -- 172.21.15.90:0/3428623319 >> 172.21.15.90:0/3428623319 conn(0x7fd8580f6920 msgr2=0x7fd8580f7660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:51.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.651+0000 7fd85e368700 1 -- 172.21.15.90:0/3428623319 shutdown_connections 2024-02-09T21:51:51.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:51.651+0000 7fd85e368700 1 -- 172.21.15.90:0/3428623319 wait complete. 2024-02-09T21:51:51.655 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-02-09T21:51:52.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:52 smithi006 ceph-mon[29258]: from='client.? 172.21.15.90:0/3428623319' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T21:51:53.231 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-09T21:51:53.232 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-09T21:51:53.481 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-09T21:51:53.527 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-09T21:51:55.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.079+0000 7ff352433700 1 -- 172.21.15.90:0/1205421107 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff34c0faf20 msgr2=0x7ff34c0fb330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:55.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.079+0000 7ff352433700 1 --2- 172.21.15.90:0/1205421107 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff34c0faf20 0x7ff34c0fb330 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff3340048c0 tx=0x7ff334009e30 comp rx=0 tx=0).stop 2024-02-09T21:51:55.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.079+0000 7ff352433700 1 -- 172.21.15.90:0/1205421107 shutdown_connections 2024-02-09T21:51:55.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.079+0000 7ff352433700 1 --2- 172.21.15.90:0/1205421107 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff34c0faf20 0x7ff34c0fb330 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:55.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.079+0000 7ff352433700 1 -- 172.21.15.90:0/1205421107 >> 172.21.15.90:0/1205421107 conn(0x7ff34c0f6920 msgr2=0x7ff34c0f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:55.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.079+0000 7ff352433700 1 -- 172.21.15.90:0/1205421107 shutdown_connections 2024-02-09T21:51:55.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.080+0000 7ff352433700 1 -- 172.21.15.90:0/1205421107 wait complete. 2024-02-09T21:51:55.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.080+0000 7ff352433700 1 Processor -- start 2024-02-09T21:51:55.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.080+0000 7ff352433700 1 -- start start 2024-02-09T21:51:55.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.080+0000 7ff352433700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff34c0faf20 0x7ff34c0fda80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:55.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.080+0000 7ff352433700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c0fbf00 con 0x7ff34c0faf20 2024-02-09T21:51:55.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.081+0000 7ff34bfff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff34c0faf20 0x7ff34c0fda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:51:55.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.081+0000 7ff34bfff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff34c0faf20 0x7ff34c0fda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.90:57540/0 (socket says 172.21.15.90:57540) 2024-02-09T21:51:55.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.081+0000 7ff34bfff700 1 -- 172.21.15.90:0/766553128 learned_addr learned my addr 172.21.15.90:0/766553128 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-09T21:51:55.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.081+0000 7ff34bfff700 1 -- 172.21.15.90:0/766553128 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff334005040 con 0x7ff34c0faf20 2024-02-09T21:51:55.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.081+0000 7ff34bfff700 1 --2- 172.21.15.90:0/766553128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff34c0faf20 0x7ff34c0fda80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff334009680 tx=0x7ff334009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:51:55.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.081+0000 7ff348ff9700 1 -- 172.21.15.90:0/766553128 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff33400b5c0 con 0x7ff34c0faf20 2024-02-09T21:51:55.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.081+0000 7ff348ff9700 1 -- 172.21.15.90:0/766553128 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff334004e60 con 0x7ff34c0faf20 2024-02-09T21:51:55.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.081+0000 7ff352433700 1 -- 172.21.15.90:0/766553128 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff34c0fc100 con 0x7ff34c0faf20 2024-02-09T21:51:55.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.081+0000 7ff348ff9700 1 -- 172.21.15.90:0/766553128 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff33401c590 con 0x7ff34c0faf20 2024-02-09T21:51:55.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.081+0000 7ff352433700 1 -- 172.21.15.90:0/766553128 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff34c0fc5f0 con 0x7ff34c0faf20 2024-02-09T21:51:55.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.082+0000 7ff348ff9700 1 -- 172.21.15.90:0/766553128 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 14) v1 ==== 43437+0+0 (secure 0 0 0) 0x7ff33401c810 con 0x7ff34c0faf20 2024-02-09T21:51:55.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.082+0000 7ff352433700 1 -- 172.21.15.90:0/766553128 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff34c106bc0 con 0x7ff34c0faf20 2024-02-09T21:51:55.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.082+0000 7ff348ff9700 1 --2- 172.21.15.90:0/766553128 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7ff3380368c0 0x7ff338038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:55.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.082+0000 7ff348ff9700 1 -- 172.21.15.90:0/766553128 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ff334049d30 con 0x7ff34c0faf20 2024-02-09T21:51:55.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.082+0000 7ff34b7fe700 1 -- 172.21.15.90:0/766553128 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7ff3380368c0 msgr2=0x7ff338038d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/3321564517 2024-02-09T21:51:55.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.082+0000 7ff34b7fe700 1 --2- 172.21.15.90:0/766553128 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7ff3380368c0 0x7ff338038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-09T21:51:55.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.087+0000 7ff348ff9700 1 -- 172.21.15.90:0/766553128 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff334013420 con 0x7ff34c0faf20 2024-02-09T21:51:55.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.283+0000 7ff34b7fe700 1 -- 172.21.15.90:0/766553128 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7ff3380368c0 msgr2=0x7ff338038d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.6:6800/3321564517 2024-02-09T21:51:55.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.283+0000 7ff34b7fe700 1 --2- 172.21.15.90:0/766553128 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7ff3380368c0 0x7ff338038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-09T21:51:55.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.342+0000 7ff352433700 1 -- 172.21.15.90:0/766553128 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff34c106bc0 con 0x7ff34c0faf20 2024-02-09T21:51:55.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.343+0000 7ff348ff9700 1 -- 172.21.15.90:0/766553128 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7ff334011b60 con 0x7ff34c0faf20 2024-02-09T21:51:55.347 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:51:55.347 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"ee26d820-c794-11ee-95b8-87774f69a715","modified":"2024-02-09T21:49:36.342734Z","created":"2024-02-09T21:49:36.342734Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi006","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3300","nonce":0},{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-09T21:51:55.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.345+0000 7ff352433700 1 -- 172.21.15.90:0/766553128 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7ff3380368c0 msgr2=0x7ff338038d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-09T21:51:55.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.345+0000 7ff352433700 1 --2- 172.21.15.90:0/766553128 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7ff3380368c0 0x7ff338038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:55.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.345+0000 7ff352433700 1 -- 172.21.15.90:0/766553128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff34c0faf20 msgr2=0x7ff34c0fda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:55.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.345+0000 7ff352433700 1 --2- 172.21.15.90:0/766553128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff34c0faf20 0x7ff34c0fda80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff334009680 tx=0x7ff334009e30 comp rx=0 tx=0).stop 2024-02-09T21:51:55.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.345+0000 7ff352433700 1 -- 172.21.15.90:0/766553128 shutdown_connections 2024-02-09T21:51:55.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.345+0000 7ff352433700 1 --2- 172.21.15.90:0/766553128 >> [v2:172.21.15.6:6800/3321564517,v1:172.21.15.6:6801/3321564517] conn(0x7ff3380368c0 0x7ff338038d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:55.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.345+0000 7ff352433700 1 --2- 172.21.15.90:0/766553128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff34c0faf20 0x7ff34c0fda80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:55.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.345+0000 7ff352433700 1 -- 172.21.15.90:0/766553128 >> 172.21.15.90:0/766553128 conn(0x7ff34c0f6920 msgr2=0x7ff34c0f7680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:55.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.345+0000 7ff352433700 1 -- 172.21.15.90:0/766553128 shutdown_connections 2024-02-09T21:51:55.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:55.345+0000 7ff352433700 1 -- 172.21.15.90:0/766553128 wait complete. 2024-02-09T21:51:55.349 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-02-09T21:51:55.464 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:55 smithi006 ceph-mon[29258]: from='client.? 172.21.15.90:0/766553128' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T21:51:56.938 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-09T21:51:56.939 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-09T21:51:57.193 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-09T21:51:57.250 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-09T21:51:57.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:57 smithi006 ceph-mon[29258]: Active manager daemon smithi006.glirxt restarted 2024-02-09T21:51:57.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:57 smithi006 ceph-mon[29258]: Activating manager daemon smithi006.glirxt 2024-02-09T21:51:57.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:57 smithi006 ceph-mon[29258]: osdmap e5: 0 total, 0 up, 0 in 2024-02-09T21:51:57.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:57 smithi006 ceph-mon[29258]: mgrmap e15: smithi006.glirxt(active, starting, since 0.00523954s) 2024-02-09T21:51:57.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:57 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata", "id": "smithi006"}]: dispatch 2024-02-09T21:51:57.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:57 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mgr metadata", "who": "smithi006.glirxt", "id": "smithi006.glirxt"}]: dispatch 2024-02-09T21:51:57.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:57 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-09T21:51:57.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:57 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-09T21:51:57.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:57 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-09T21:51:57.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:57 smithi006 ceph-mon[29258]: Manager daemon smithi006.glirxt is now available 2024-02-09T21:51:57.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:57 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:51:57.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:57 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:51:57.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:57 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:51:57.659 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:57 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:51:57.659 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:57 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi006.glirxt/mirror_snapshot_schedule"}]: dispatch 2024-02-09T21:51:57.659 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:57 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi006.glirxt/trash_purge_schedule"}]: dispatch 2024-02-09T21:51:58.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:58 smithi006 ceph-mon[29258]: [09/Feb/2024:21:51:57] ENGINE Bus STARTING 2024-02-09T21:51:58.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:58 smithi006 ceph-mon[29258]: [09/Feb/2024:21:51:57] ENGINE Serving on https://172.21.15.6:7150 2024-02-09T21:51:58.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:58 smithi006 ceph-mon[29258]: [09/Feb/2024:21:51:57] ENGINE Bus STARTED 2024-02-09T21:51:58.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:58 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:51:58.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:58 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:51:58.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:58 smithi006 ceph-mon[29258]: mgrmap e16: smithi006.glirxt(active, since 1.01009s) 2024-02-09T21:51:58.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:58 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:51:58.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:58.754+0000 7f67a978d700 1 -- 172.21.15.90:0/3745776346 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67a40fbab0 msgr2=0x7f67a40fbec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:58.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:58.754+0000 7f67a978d700 1 --2- 172.21.15.90:0/3745776346 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67a40fbab0 0x7f67a40fbec0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f67940048c0 tx=0x7f6794009e30 comp rx=0 tx=0).stop 2024-02-09T21:51:58.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:58.754+0000 7f67a978d700 1 -- 172.21.15.90:0/3745776346 shutdown_connections 2024-02-09T21:51:58.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:58.754+0000 7f67a978d700 1 --2- 172.21.15.90:0/3745776346 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67a40fbab0 0x7f67a40fbec0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:58.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:58.754+0000 7f67a978d700 1 -- 172.21.15.90:0/3745776346 >> 172.21.15.90:0/3745776346 conn(0x7f67a40f7210 msgr2=0x7f67a40f9640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:58.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:58.754+0000 7f67a978d700 1 -- 172.21.15.90:0/3745776346 shutdown_connections 2024-02-09T21:51:58.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:58.754+0000 7f67a978d700 1 -- 172.21.15.90:0/3745776346 wait complete. 2024-02-09T21:51:58.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:58.755+0000 7f67a978d700 1 Processor -- start 2024-02-09T21:51:58.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:58.755+0000 7f67a978d700 1 -- start start 2024-02-09T21:51:58.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:58.755+0000 7f67a978d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67a40fbab0 0x7f67a410ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:58.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:58.755+0000 7f67a978d700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a410d3d0 con 0x7f67a40fbab0 2024-02-09T21:51:58.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:58.755+0000 7f67a2ffd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67a40fbab0 0x7f67a410ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:51:58.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:58.755+0000 7f67a2ffd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67a40fbab0 0x7f67a410ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.90:54572/0 (socket says 172.21.15.90:54572) 2024-02-09T21:51:58.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:58.755+0000 7f67a2ffd700 1 -- 172.21.15.90:0/3126273264 learned_addr learned my addr 172.21.15.90:0/3126273264 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-09T21:51:58.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:58.755+0000 7f67a2ffd700 1 -- 172.21.15.90:0/3126273264 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6794005040 con 0x7f67a40fbab0 2024-02-09T21:51:58.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:58.756+0000 7f67a2ffd700 1 --2- 172.21.15.90:0/3126273264 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67a40fbab0 0x7f67a410ce90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6794009680 tx=0x7f6794009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:51:58.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:58.756+0000 7f678b7fe700 1 -- 172.21.15.90:0/3126273264 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f679400b5c0 con 0x7f67a40fbab0 2024-02-09T21:51:58.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:58.756+0000 7f678b7fe700 1 -- 172.21.15.90:0/3126273264 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6794004e60 con 0x7f67a40fbab0 2024-02-09T21:51:58.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:58.756+0000 7f678b7fe700 1 -- 172.21.15.90:0/3126273264 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f679401c590 con 0x7f67a40fbab0 2024-02-09T21:51:58.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:58.756+0000 7f67a978d700 1 -- 172.21.15.90:0/3126273264 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67a410d5d0 con 0x7f67a40fbab0 2024-02-09T21:51:58.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:58.756+0000 7f67a978d700 1 -- 172.21.15.90:0/3126273264 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67a410e270 con 0x7f67a40fbab0 2024-02-09T21:51:58.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:58.757+0000 7f678b7fe700 1 -- 172.21.15.90:0/3126273264 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 16) v1 ==== 43289+0+0 (secure 0 0 0) 0x7f679401c810 con 0x7f67a40fbab0 2024-02-09T21:51:58.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:58.757+0000 7f678b7fe700 1 --2- 172.21.15.90:0/3126273264 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f678c036700 0x7f678c038bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:51:58.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:58.757+0000 7f678b7fe700 1 -- 172.21.15.90:0/3126273264 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f6794049c80 con 0x7f67a40fbab0 2024-02-09T21:51:58.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:58.757+0000 7f67a978d700 1 -- 172.21.15.90:0/3126273264 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67a405f5c0 con 0x7f67a40fbab0 2024-02-09T21:51:58.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:58.757+0000 7f67a27fc700 1 --2- 172.21.15.90:0/3126273264 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f678c036700 0x7f678c038bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:51:58.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:58.758+0000 7f67a27fc700 1 --2- 172.21.15.90:0/3126273264 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f678c036700 0x7f678c038bb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6798009990 tx=0x7f6798006e40 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:51:58.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:58.762+0000 7f678b7fe700 1 -- 172.21.15.90:0/3126273264 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6794023300 con 0x7f67a40fbab0 2024-02-09T21:51:59.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:59.038+0000 7f67a978d700 1 -- 172.21.15.90:0/3126273264 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f67a410ed40 con 0x7f67a40fbab0 2024-02-09T21:51:59.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:59.038+0000 7f678b7fe700 1 -- 172.21.15.90:0/3126273264 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f6794034090 con 0x7f67a40fbab0 2024-02-09T21:51:59.043 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:51:59.043 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"ee26d820-c794-11ee-95b8-87774f69a715","modified":"2024-02-09T21:49:36.342734Z","created":"2024-02-09T21:49:36.342734Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi006","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3300","nonce":0},{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-09T21:51:59.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:59.041+0000 7f67a978d700 1 -- 172.21.15.90:0/3126273264 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f678c036700 msgr2=0x7f678c038bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:59.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:59.041+0000 7f67a978d700 1 --2- 172.21.15.90:0/3126273264 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f678c036700 0x7f678c038bb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6798009990 tx=0x7f6798006e40 comp rx=0 tx=0).stop 2024-02-09T21:51:59.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:59.041+0000 7f67a978d700 1 -- 172.21.15.90:0/3126273264 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67a40fbab0 msgr2=0x7f67a410ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:51:59.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:59.041+0000 7f67a978d700 1 --2- 172.21.15.90:0/3126273264 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67a40fbab0 0x7f67a410ce90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6794009680 tx=0x7f6794009e30 comp rx=0 tx=0).stop 2024-02-09T21:51:59.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:59.041+0000 7f67a978d700 1 -- 172.21.15.90:0/3126273264 shutdown_connections 2024-02-09T21:51:59.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:59.041+0000 7f67a978d700 1 --2- 172.21.15.90:0/3126273264 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f678c036700 0x7f678c038bb0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:59.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:59.041+0000 7f67a978d700 1 --2- 172.21.15.90:0/3126273264 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67a40fbab0 0x7f67a410ce90 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:51:59.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:59.041+0000 7f67a978d700 1 -- 172.21.15.90:0/3126273264 >> 172.21.15.90:0/3126273264 conn(0x7f67a40f7210 msgr2=0x7f67a4066390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:51:59.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:59.041+0000 7f67a978d700 1 -- 172.21.15.90:0/3126273264 shutdown_connections 2024-02-09T21:51:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:51:59.041+0000 7f67a978d700 1 -- 172.21.15.90:0/3126273264 wait complete. 2024-02-09T21:51:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-02-09T21:52:00.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:59 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:00.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:51:59 smithi006 ceph-mon[29258]: from='client.? 172.21.15.90:0/3126273264' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T21:52:00.649 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-09T21:52:00.649 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-09T21:52:00.987 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-09T21:52:01.048 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-09T21:52:01.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:00 smithi006 ceph-mon[29258]: mgrmap e17: smithi006.glirxt(active, since 2s) 2024-02-09T21:52:02.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:02.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:02.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:02.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:02.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config rm", "who": "osd/host:smithi006", "name": "osd_memory_target"}]: dispatch 2024-02-09T21:52:03.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.252+0000 7facf0d04700 1 -- 172.21.15.90:0/4019869073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7facec0702b0 msgr2=0x7facec070680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:52:03.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.253+0000 7facf0d04700 1 --2- 172.21.15.90:0/4019869073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7facec0702b0 0x7facec070680 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7facdc0048c0 tx=0x7facdc009e30 comp rx=0 tx=0).stop 2024-02-09T21:52:03.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.253+0000 7facf0d04700 1 -- 172.21.15.90:0/4019869073 shutdown_connections 2024-02-09T21:52:03.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.253+0000 7facf0d04700 1 --2- 172.21.15.90:0/4019869073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7facec0702b0 0x7facec070680 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:52:03.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.253+0000 7facf0d04700 1 -- 172.21.15.90:0/4019869073 >> 172.21.15.90:0/4019869073 conn(0x7facec0f8760 msgr2=0x7facec0fab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:52:03.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.253+0000 7facf0d04700 1 -- 172.21.15.90:0/4019869073 shutdown_connections 2024-02-09T21:52:03.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.253+0000 7facf0d04700 1 -- 172.21.15.90:0/4019869073 wait complete. 2024-02-09T21:52:03.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.253+0000 7facf0d04700 1 Processor -- start 2024-02-09T21:52:03.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.253+0000 7facf0d04700 1 -- start start 2024-02-09T21:52:03.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.253+0000 7facf0d04700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7facec0702b0 0x7facec1151e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:52:03.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.253+0000 7facf0d04700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facec115720 con 0x7facec0702b0 2024-02-09T21:52:03.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.254+0000 7facea59c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7facec0702b0 0x7facec1151e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:52:03.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.254+0000 7facea59c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7facec0702b0 0x7facec1151e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.90:54600/0 (socket says 172.21.15.90:54600) 2024-02-09T21:52:03.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.254+0000 7facea59c700 1 -- 172.21.15.90:0/691675021 learned_addr learned my addr 172.21.15.90:0/691675021 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-09T21:52:03.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.254+0000 7facea59c700 1 -- 172.21.15.90:0/691675021 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facdc005040 con 0x7facec0702b0 2024-02-09T21:52:03.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.254+0000 7facea59c700 1 --2- 172.21.15.90:0/691675021 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7facec0702b0 0x7facec1151e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7facdc009300 tx=0x7facdc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:52:03.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.254+0000 7facdaffd700 1 -- 172.21.15.90:0/691675021 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facdc004e60 con 0x7facec0702b0 2024-02-09T21:52:03.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.254+0000 7facdaffd700 1 -- 172.21.15.90:0/691675021 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7facdc013420 con 0x7facec0702b0 2024-02-09T21:52:03.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.254+0000 7facf0d04700 1 -- 172.21.15.90:0/691675021 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facec115920 con 0x7facec0702b0 2024-02-09T21:52:03.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.254+0000 7facdaffd700 1 -- 172.21.15.90:0/691675021 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facdc01c5b0 con 0x7facec0702b0 2024-02-09T21:52:03.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.254+0000 7facf0d04700 1 -- 172.21.15.90:0/691675021 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facec115d90 con 0x7facec0702b0 2024-02-09T21:52:03.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.255+0000 7facdaffd700 1 -- 172.21.15.90:0/691675021 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 17) v1 ==== 43479+0+0 (secure 0 0 0) 0x7facdc01c7a0 con 0x7facec0702b0 2024-02-09T21:52:03.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.255+0000 7facf0d04700 1 -- 172.21.15.90:0/691675021 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facec05f5c0 con 0x7facec0702b0 2024-02-09T21:52:03.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.255+0000 7facdaffd700 1 --2- 172.21.15.90:0/691675021 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7facd40369b0 0x7facd4038e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:52:03.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.256+0000 7face9d9b700 1 --2- 172.21.15.90:0/691675021 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7facd40369b0 0x7facd4038e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:52:03.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.256+0000 7facdaffd700 1 -- 172.21.15.90:0/691675021 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7facdc049ab0 con 0x7facec0702b0 2024-02-09T21:52:03.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.256+0000 7face9d9b700 1 --2- 172.21.15.90:0/691675021 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7facd40369b0 0x7facd4038e60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7face0009990 tx=0x7face0006e40 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:52:03.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.260+0000 7facdaffd700 1 -- 172.21.15.90:0/691675021 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7facdc011d40 con 0x7facec0702b0 2024-02-09T21:52:03.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.530+0000 7facf0d04700 1 -- 172.21.15.90:0/691675021 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7facec1090f0 con 0x7facec0702b0 2024-02-09T21:52:03.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.530+0000 7facdaffd700 1 -- 172.21.15.90:0/691675021 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7facdc0116f0 con 0x7facec0702b0 2024-02-09T21:52:03.536 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:52:03.536 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"ee26d820-c794-11ee-95b8-87774f69a715","modified":"2024-02-09T21:49:36.342734Z","created":"2024-02-09T21:49:36.342734Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi006","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3300","nonce":0},{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-09T21:52:03.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.532+0000 7facf0d04700 1 -- 172.21.15.90:0/691675021 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7facd40369b0 msgr2=0x7facd4038e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:52:03.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.532+0000 7facf0d04700 1 --2- 172.21.15.90:0/691675021 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7facd40369b0 0x7facd4038e60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7face0009990 tx=0x7face0006e40 comp rx=0 tx=0).stop 2024-02-09T21:52:03.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.532+0000 7facf0d04700 1 -- 172.21.15.90:0/691675021 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7facec0702b0 msgr2=0x7facec1151e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:52:03.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.532+0000 7facf0d04700 1 --2- 172.21.15.90:0/691675021 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7facec0702b0 0x7facec1151e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7facdc009300 tx=0x7facdc009e30 comp rx=0 tx=0).stop 2024-02-09T21:52:03.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.533+0000 7facf0d04700 1 -- 172.21.15.90:0/691675021 shutdown_connections 2024-02-09T21:52:03.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.533+0000 7facf0d04700 1 --2- 172.21.15.90:0/691675021 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7facd40369b0 0x7facd4038e60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:52:03.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.533+0000 7facf0d04700 1 --2- 172.21.15.90:0/691675021 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7facec0702b0 0x7facec1151e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:52:03.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.533+0000 7facf0d04700 1 -- 172.21.15.90:0/691675021 >> 172.21.15.90:0/691675021 conn(0x7facec0f8760 msgr2=0x7facec103780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:52:03.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.533+0000 7facf0d04700 1 -- 172.21.15.90:0/691675021 shutdown_connections 2024-02-09T21:52:03.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:03.533+0000 7facf0d04700 1 -- 172.21.15.90:0/691675021 wait complete. 2024-02-09T21:52:03.540 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-02-09T21:52:03.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:03 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:03.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:03 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:03.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:03 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:03.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:03 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:03.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:03 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config rm", "who": "osd/host:smithi090", "name": "osd_memory_target"}]: dispatch 2024-02-09T21:52:03.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:03 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:52:03.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:03 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:52:03.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:03 smithi006 ceph-mon[29258]: Updating smithi006:/etc/ceph/ceph.conf 2024-02-09T21:52:03.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:03 smithi006 ceph-mon[29258]: Updating smithi090:/etc/ceph/ceph.conf 2024-02-09T21:52:03.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:03 smithi006 ceph-mon[29258]: Updating smithi006:/var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T21:52:03.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:03 smithi006 ceph-mon[29258]: Updating smithi090:/var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T21:52:03.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:03 smithi006 ceph-mon[29258]: Updating smithi006:/etc/ceph/ceph.client.admin.keyring 2024-02-09T21:52:03.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:03 smithi006 ceph-mon[29258]: Updating smithi090:/etc/ceph/ceph.client.admin.keyring 2024-02-09T21:52:03.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:03 smithi006 ceph-mon[29258]: Updating smithi006:/var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-09T21:52:03.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:03 smithi006 ceph-mon[29258]: Updating smithi090:/var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-09T21:52:03.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:03 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:03.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:03 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:03.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:03 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:03.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:03 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:03.659 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:03 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:03.659 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:03 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi090", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-09T21:52:03.659 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:03 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi090", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-09T21:52:03.659 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:03 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:52:04.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:04 smithi006 ceph-mon[29258]: Deploying daemon crash.smithi090 on smithi090 2024-02-09T21:52:04.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:04 smithi006 ceph-mon[29258]: from='client.? 172.21.15.90:0/691675021' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T21:52:05.591 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-09T21:52:05.592 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-09T21:52:06.414 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T21:52:07.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:07 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:07.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:07 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:07.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:07 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:07.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:07 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:07.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:07 smithi006 ceph-mon[29258]: Deploying daemon node-exporter.smithi090 on smithi090 2024-02-09T21:52:07.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:07 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:07.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:07.810+0000 7fcc7b7c5700 1 -- 172.21.15.90:0/35455356 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc74102bd0 msgr2=0x7fcc74102fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:52:07.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:07.810+0000 7fcc7b7c5700 1 --2- 172.21.15.90:0/35455356 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc74102bd0 0x7fcc74102fa0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcc700048c0 tx=0x7fcc70009e30 comp rx=0 tx=0).stop 2024-02-09T21:52:07.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:07.810+0000 7fcc7b7c5700 1 -- 172.21.15.90:0/35455356 shutdown_connections 2024-02-09T21:52:07.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:07.810+0000 7fcc7b7c5700 1 --2- 172.21.15.90:0/35455356 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc74102bd0 0x7fcc74102fa0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:52:07.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:07.810+0000 7fcc7b7c5700 1 -- 172.21.15.90:0/35455356 >> 172.21.15.90:0/35455356 conn(0x7fcc740f8740 msgr2=0x7fcc740fab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:52:07.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:07.810+0000 7fcc7b7c5700 1 -- 172.21.15.90:0/35455356 shutdown_connections 2024-02-09T21:52:07.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:07.810+0000 7fcc7b7c5700 1 -- 172.21.15.90:0/35455356 wait complete. 2024-02-09T21:52:07.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:07.811+0000 7fcc7b7c5700 1 Processor -- start 2024-02-09T21:52:07.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:07.811+0000 7fcc7b7c5700 1 -- start start 2024-02-09T21:52:07.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:07.811+0000 7fcc7b7c5700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc74102bd0 0x7fcc74070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:52:07.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:07.811+0000 7fcc7b7c5700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc740707d0 con 0x7fcc74102bd0 2024-02-09T21:52:07.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:07.812+0000 7fcc79561700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc74102bd0 0x7fcc74070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:52:07.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:07.812+0000 7fcc79561700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc74102bd0 0x7fcc74070290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.90:53542/0 (socket says 172.21.15.90:53542) 2024-02-09T21:52:07.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:07.812+0000 7fcc79561700 1 -- 172.21.15.90:0/1966865236 learned_addr learned my addr 172.21.15.90:0/1966865236 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-09T21:52:07.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:07.812+0000 7fcc79561700 1 -- 172.21.15.90:0/1966865236 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc70005040 con 0x7fcc74102bd0 2024-02-09T21:52:07.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:07.812+0000 7fcc79561700 1 --2- 172.21.15.90:0/1966865236 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc74102bd0 0x7fcc74070290 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcc70009300 tx=0x7fcc70009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:52:07.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:07.812+0000 7fcc69ffb700 1 -- 172.21.15.90:0/1966865236 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc70004e60 con 0x7fcc74102bd0 2024-02-09T21:52:07.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:07.812+0000 7fcc69ffb700 1 -- 172.21.15.90:0/1966865236 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcc70013420 con 0x7fcc74102bd0 2024-02-09T21:52:07.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:07.812+0000 7fcc7b7c5700 1 -- 172.21.15.90:0/1966865236 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc740709d0 con 0x7fcc74102bd0 2024-02-09T21:52:07.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:07.812+0000 7fcc69ffb700 1 -- 172.21.15.90:0/1966865236 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc7001c5b0 con 0x7fcc74102bd0 2024-02-09T21:52:07.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:07.813+0000 7fcc7b7c5700 1 -- 172.21.15.90:0/1966865236 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc74070ec0 con 0x7fcc74102bd0 2024-02-09T21:52:07.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:07.813+0000 7fcc7b7c5700 1 -- 172.21.15.90:0/1966865236 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc74071100 con 0x7fcc74102bd0 2024-02-09T21:52:07.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:07.814+0000 7fcc69ffb700 1 -- 172.21.15.90:0/1966865236 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 17) v1 ==== 43479+0+0 (secure 0 0 0) 0x7fcc70013a90 con 0x7fcc74102bd0 2024-02-09T21:52:07.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:07.814+0000 7fcc69ffb700 1 --2- 172.21.15.90:0/1966865236 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fcc60036960 0x7fcc60038e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:52:07.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:07.814+0000 7fcc69ffb700 1 -- 172.21.15.90:0/1966865236 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fcc70049bf0 con 0x7fcc74102bd0 2024-02-09T21:52:07.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:07.814+0000 7fcc78d60700 1 --2- 172.21.15.90:0/1966865236 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fcc60036960 0x7fcc60038e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:52:07.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:07.815+0000 7fcc78d60700 1 --2- 172.21.15.90:0/1966865236 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fcc60036960 0x7fcc60038e10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcc64009990 tx=0x7fcc64006e40 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:52:07.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:07.819+0000 7fcc69ffb700 1 -- 172.21.15.90:0/1966865236 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fcc74071100 con 0x7fcc74102bd0 2024-02-09T21:52:08.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:08.095+0000 7fcc7b7c5700 1 -- 172.21.15.90:0/1966865236 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcc74071100 con 0x7fcc74102bd0 2024-02-09T21:52:08.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:08.096+0000 7fcc69ffb700 1 -- 172.21.15.90:0/1966865236 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7fcc74071100 con 0x7fcc74102bd0 2024-02-09T21:52:08.100 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:52:08.100 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"ee26d820-c794-11ee-95b8-87774f69a715","modified":"2024-02-09T21:49:36.342734Z","created":"2024-02-09T21:49:36.342734Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi006","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3300","nonce":0},{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-09T21:52:08.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:08.098+0000 7fcc7b7c5700 1 -- 172.21.15.90:0/1966865236 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fcc60036960 msgr2=0x7fcc60038e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:52:08.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:08.098+0000 7fcc7b7c5700 1 --2- 172.21.15.90:0/1966865236 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fcc60036960 0x7fcc60038e10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcc64009990 tx=0x7fcc64006e40 comp rx=0 tx=0).stop 2024-02-09T21:52:08.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:08.098+0000 7fcc7b7c5700 1 -- 172.21.15.90:0/1966865236 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc74102bd0 msgr2=0x7fcc74070290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:52:08.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:08.098+0000 7fcc7b7c5700 1 --2- 172.21.15.90:0/1966865236 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc74102bd0 0x7fcc74070290 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcc70009300 tx=0x7fcc70009e30 comp rx=0 tx=0).stop 2024-02-09T21:52:08.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:08.098+0000 7fcc7b7c5700 1 -- 172.21.15.90:0/1966865236 shutdown_connections 2024-02-09T21:52:08.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:08.098+0000 7fcc7b7c5700 1 --2- 172.21.15.90:0/1966865236 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fcc60036960 0x7fcc60038e10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:52:08.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:08.098+0000 7fcc7b7c5700 1 --2- 172.21.15.90:0/1966865236 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc74102bd0 0x7fcc74070290 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:52:08.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:08.098+0000 7fcc7b7c5700 1 -- 172.21.15.90:0/1966865236 >> 172.21.15.90:0/1966865236 conn(0x7fcc740f8740 msgr2=0x7fcc740fa5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:52:08.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:08.098+0000 7fcc7b7c5700 1 -- 172.21.15.90:0/1966865236 shutdown_connections 2024-02-09T21:52:08.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:08.098+0000 7fcc7b7c5700 1 -- 172.21.15.90:0/1966865236 wait complete. 2024-02-09T21:52:08.103 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-02-09T21:52:08.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:08 smithi006 ceph-mon[29258]: from='client.? 172.21.15.90:0/1966865236' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T21:52:09.854 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-09T21:52:09.854 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-09T21:52:10.196 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T21:52:12.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.447+0000 7f64915a7700 1 -- 172.21.15.90:0/2002611572 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f648c102bd0 msgr2=0x7f648c102fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:52:12.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.447+0000 7f64915a7700 1 --2- 172.21.15.90:0/2002611572 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f648c102bd0 0x7f648c102fa0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f647c0048c0 tx=0x7f647c009e30 comp rx=0 tx=0).stop 2024-02-09T21:52:12.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.447+0000 7f64915a7700 1 -- 172.21.15.90:0/2002611572 shutdown_connections 2024-02-09T21:52:12.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.447+0000 7f64915a7700 1 --2- 172.21.15.90:0/2002611572 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f648c102bd0 0x7f648c102fa0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:52:12.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.447+0000 7f64915a7700 1 -- 172.21.15.90:0/2002611572 >> 172.21.15.90:0/2002611572 conn(0x7f648c0f8740 msgr2=0x7f648c0fab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:52:12.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.447+0000 7f64915a7700 1 -- 172.21.15.90:0/2002611572 shutdown_connections 2024-02-09T21:52:12.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.447+0000 7f64915a7700 1 -- 172.21.15.90:0/2002611572 wait complete. 2024-02-09T21:52:12.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.448+0000 7f64915a7700 1 Processor -- start 2024-02-09T21:52:12.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.448+0000 7f64915a7700 1 -- start start 2024-02-09T21:52:12.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.448+0000 7f64915a7700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f648c102bd0 0x7f648c112f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:52:12.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.448+0000 7f64915a7700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648c1134d0 con 0x7f648c102bd0 2024-02-09T21:52:12.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.448+0000 7f648affd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f648c102bd0 0x7f648c112f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:52:12.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.448+0000 7f648affd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f648c102bd0 0x7f648c112f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.90:53552/0 (socket says 172.21.15.90:53552) 2024-02-09T21:52:12.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.448+0000 7f648affd700 1 -- 172.21.15.90:0/1773276071 learned_addr learned my addr 172.21.15.90:0/1773276071 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-09T21:52:12.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.449+0000 7f648affd700 1 -- 172.21.15.90:0/1773276071 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f647c005040 con 0x7f648c102bd0 2024-02-09T21:52:12.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.449+0000 7f648affd700 1 --2- 172.21.15.90:0/1773276071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f648c102bd0 0x7f648c112f90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f647c009300 tx=0x7f647c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:52:12.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.449+0000 7f64737fe700 1 -- 172.21.15.90:0/1773276071 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f647c004e60 con 0x7f648c102bd0 2024-02-09T21:52:12.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.449+0000 7f64915a7700 1 -- 172.21.15.90:0/1773276071 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f648c1136d0 con 0x7f648c102bd0 2024-02-09T21:52:12.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.449+0000 7f64737fe700 1 -- 172.21.15.90:0/1773276071 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f647c013420 con 0x7f648c102bd0 2024-02-09T21:52:12.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.449+0000 7f64737fe700 1 -- 172.21.15.90:0/1773276071 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f647c01c5b0 con 0x7f648c102bd0 2024-02-09T21:52:12.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.449+0000 7f64915a7700 1 -- 172.21.15.90:0/1773276071 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f648c113bc0 con 0x7f648c102bd0 2024-02-09T21:52:12.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.450+0000 7f64915a7700 1 -- 172.21.15.90:0/1773276071 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f648c05f5c0 con 0x7f648c102bd0 2024-02-09T21:52:12.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.450+0000 7f64737fe700 1 -- 172.21.15.90:0/1773276071 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 17) v1 ==== 43479+0+0 (secure 0 0 0) 0x7f647c009600 con 0x7f648c102bd0 2024-02-09T21:52:12.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.450+0000 7f64737fe700 1 --2- 172.21.15.90:0/1773276071 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f6474036960 0x7f6474038e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:52:12.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.450+0000 7f64737fe700 1 -- 172.21.15.90:0/1773276071 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f647c049b80 con 0x7f648c102bd0 2024-02-09T21:52:12.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.450+0000 7f648a7fc700 1 --2- 172.21.15.90:0/1773276071 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f6474036960 0x7f6474038e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:52:12.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.451+0000 7f648a7fc700 1 --2- 172.21.15.90:0/1773276071 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f6474036960 0x7f6474038e10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6480009990 tx=0x7f6480006e40 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:52:12.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.455+0000 7f64737fe700 1 -- 172.21.15.90:0/1773276071 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f647c025400 con 0x7f648c102bd0 2024-02-09T21:52:12.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.723+0000 7f64915a7700 1 -- 172.21.15.90:0/1773276071 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f648c10e700 con 0x7f648c102bd0 2024-02-09T21:52:12.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.723+0000 7f64737fe700 1 -- 172.21.15.90:0/1773276071 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f647c011370 con 0x7f648c102bd0 2024-02-09T21:52:12.727 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:52:12.727 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"ee26d820-c794-11ee-95b8-87774f69a715","modified":"2024-02-09T21:49:36.342734Z","created":"2024-02-09T21:49:36.342734Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi006","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3300","nonce":0},{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-09T21:52:12.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.725+0000 7f64915a7700 1 -- 172.21.15.90:0/1773276071 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f6474036960 msgr2=0x7f6474038e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:52:12.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.725+0000 7f64915a7700 1 --2- 172.21.15.90:0/1773276071 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f6474036960 0x7f6474038e10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6480009990 tx=0x7f6480006e40 comp rx=0 tx=0).stop 2024-02-09T21:52:12.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.725+0000 7f64915a7700 1 -- 172.21.15.90:0/1773276071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f648c102bd0 msgr2=0x7f648c112f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:52:12.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.725+0000 7f64915a7700 1 --2- 172.21.15.90:0/1773276071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f648c102bd0 0x7f648c112f90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f647c009300 tx=0x7f647c009e30 comp rx=0 tx=0).stop 2024-02-09T21:52:12.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.725+0000 7f64915a7700 1 -- 172.21.15.90:0/1773276071 shutdown_connections 2024-02-09T21:52:12.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.725+0000 7f64915a7700 1 --2- 172.21.15.90:0/1773276071 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f6474036960 0x7f6474038e10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:52:12.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.726+0000 7f64915a7700 1 --2- 172.21.15.90:0/1773276071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f648c102bd0 0x7f648c112f90 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:52:12.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.726+0000 7f64915a7700 1 -- 172.21.15.90:0/1773276071 >> 172.21.15.90:0/1773276071 conn(0x7f648c0f8740 msgr2=0x7f648c0fa630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:52:12.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.726+0000 7f64915a7700 1 -- 172.21.15.90:0/1773276071 shutdown_connections 2024-02-09T21:52:12.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:12.726+0000 7f64915a7700 1 -- 172.21.15.90:0/1773276071 wait complete. 2024-02-09T21:52:12.730 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-02-09T21:52:12.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:12 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:12.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:12 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:12.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:12 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:12.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:12 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:12.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:12 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi090.csxosj", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-09T21:52:12.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:12 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi090.csxosj", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-09T21:52:12.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:12 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-09T21:52:12.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:12 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:52:12.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:12 smithi006 ceph-mon[29258]: Deploying daemon mgr.smithi090.csxosj on smithi090 2024-02-09T21:52:12.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:12 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:52:12.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:12 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:13.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:13 smithi006 ceph-mon[29258]: from='client.? 172.21.15.90:0/1773276071' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T21:52:15.317 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-09T21:52:15.318 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-09T21:52:16.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:15 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:16.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:15 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:16.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:15 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:16.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:15 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:16.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:15 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T21:52:16.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:15 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:52:16.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:15 smithi006 ceph-mon[29258]: Deploying daemon mon.smithi090 on smithi090 2024-02-09T21:52:17.252 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi090/config 2024-02-09T21:52:18.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:17 smithi006 ceph-mon[29258]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:18.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:17 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:19.004 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:18 smithi090 systemd[1]: Started Ceph mon.smithi090 for ee26d820-c794-11ee-95b8-87774f69a715. 2024-02-09T21:52:24.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:23 smithi006 ceph-mon[29258]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:24.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:23 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata", "id": "smithi006"}]: dispatch 2024-02-09T21:52:24.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:23 smithi006 ceph-mon[29258]: mon.smithi006 calling monitor election 2024-02-09T21:52:24.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:23 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-02-09T21:52:24.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:23 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-02-09T21:52:24.409 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:23 smithi006 ceph-mon[29258]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:24.409 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:23 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-02-09T21:52:24.409 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:23 smithi006 ceph-mon[29258]: mon.smithi090 calling monitor election 2024-02-09T21:52:24.409 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:23 smithi006 ceph-mon[29258]: from='mgr.? 172.21.15.90:0/1633341337' entity='mgr.smithi090.csxosj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi090.csxosj/crt"}]: dispatch 2024-02-09T21:52:24.409 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:23 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-02-09T21:52:24.409 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:23 smithi006 ceph-mon[29258]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:24.409 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:23 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-02-09T21:52:24.409 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:23 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-02-09T21:52:24.410 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:23 smithi006 ceph-mon[29258]: mon.smithi006 is new leader, mons smithi006,smithi090 in quorum (ranks 0,1) 2024-02-09T21:52:24.410 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:23 smithi006 ceph-mon[29258]: monmap e2: 2 mons at {smithi006=[v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0],smithi090=[v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-09T21:52:24.410 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:23 smithi006 ceph-mon[29258]: fsmap 2024-02-09T21:52:24.410 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:23 smithi006 ceph-mon[29258]: osdmap e5: 0 total, 0 up, 0 in 2024-02-09T21:52:24.410 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:23 smithi006 ceph-mon[29258]: mgrmap e17: smithi006.glirxt(active, since 27s) 2024-02-09T21:52:24.410 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:23 smithi006 ceph-mon[29258]: Standby manager daemon smithi090.csxosj started 2024-02-09T21:52:24.410 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:23 smithi006 ceph-mon[29258]: from='mgr.? 172.21.15.90:0/1633341337' entity='mgr.smithi090.csxosj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-09T21:52:24.410 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:23 smithi006 ceph-mon[29258]: overall HEALTH_OK 2024-02-09T21:52:24.410 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:23 smithi006 ceph-mon[29258]: from='mgr.? 172.21.15.90:0/1633341337' entity='mgr.smithi090.csxosj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi090.csxosj/key"}]: dispatch 2024-02-09T21:52:24.411 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:23 smithi006 ceph-mon[29258]: from='mgr.? 172.21.15.90:0/1633341337' entity='mgr.smithi090.csxosj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-09T21:52:24.411 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:23 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:24.411 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:23 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:24.411 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:23 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:24.411 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:23 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:24.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:23 smithi090 ceph-mon[36996]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:24.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:23 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata", "id": "smithi006"}]: dispatch 2024-02-09T21:52:24.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:23 smithi090 ceph-mon[36996]: mon.smithi006 calling monitor election 2024-02-09T21:52:24.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:23 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-02-09T21:52:24.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:23 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-02-09T21:52:24.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:23 smithi090 ceph-mon[36996]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:24.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:23 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-02-09T21:52:24.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:23 smithi090 ceph-mon[36996]: mon.smithi090 calling monitor election 2024-02-09T21:52:24.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:23 smithi090 ceph-mon[36996]: from='mgr.? 172.21.15.90:0/1633341337' entity='mgr.smithi090.csxosj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi090.csxosj/crt"}]: dispatch 2024-02-09T21:52:24.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:23 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-02-09T21:52:24.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:23 smithi090 ceph-mon[36996]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:24.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:23 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-02-09T21:52:24.456 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:23 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-02-09T21:52:24.456 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:23 smithi090 ceph-mon[36996]: mon.smithi006 is new leader, mons smithi006,smithi090 in quorum (ranks 0,1) 2024-02-09T21:52:24.456 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:23 smithi090 ceph-mon[36996]: monmap e2: 2 mons at {smithi006=[v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0],smithi090=[v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-09T21:52:24.456 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:23 smithi090 ceph-mon[36996]: fsmap 2024-02-09T21:52:24.456 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:23 smithi090 ceph-mon[36996]: osdmap e5: 0 total, 0 up, 0 in 2024-02-09T21:52:24.456 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:23 smithi090 ceph-mon[36996]: mgrmap e17: smithi006.glirxt(active, since 27s) 2024-02-09T21:52:24.456 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:23 smithi090 ceph-mon[36996]: Standby manager daemon smithi090.csxosj started 2024-02-09T21:52:24.456 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:23 smithi090 ceph-mon[36996]: from='mgr.? 172.21.15.90:0/1633341337' entity='mgr.smithi090.csxosj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-09T21:52:24.456 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:23 smithi090 ceph-mon[36996]: overall HEALTH_OK 2024-02-09T21:52:24.456 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:23 smithi090 ceph-mon[36996]: from='mgr.? 172.21.15.90:0/1633341337' entity='mgr.smithi090.csxosj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi090.csxosj/key"}]: dispatch 2024-02-09T21:52:24.456 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:23 smithi090 ceph-mon[36996]: from='mgr.? 172.21.15.90:0/1633341337' entity='mgr.smithi090.csxosj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-09T21:52:24.456 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:23 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:24.456 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:23 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:24.456 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:23 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:24.456 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:23 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:25.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:24 smithi006 ceph-mon[29258]: mgrmap e18: smithi006.glirxt(active, since 27s), standbys: smithi090.csxosj 2024-02-09T21:52:25.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:24 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:25.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:24 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mgr metadata", "who": "smithi090.csxosj", "id": "smithi090.csxosj"}]: dispatch 2024-02-09T21:52:25.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:24 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:52:25.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:24 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-02-09T21:52:25.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.450+0000 7f806e924700 1 -- 172.21.15.90:0/105699411 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80500061f0 msgr2=0x7f8050008650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:52:25.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.450+0000 7f806e924700 1 --2- 172.21.15.90:0/105699411 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80500061f0 0x7f8050008650 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f80680fccb0 tx=0x7f8058002040 comp rx=0 tx=0).stop 2024-02-09T21:52:25.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.450+0000 7f806e924700 1 -- 172.21.15.90:0/105699411 shutdown_connections 2024-02-09T21:52:25.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.450+0000 7f806e924700 1 --2- 172.21.15.90:0/105699411 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80500061f0 0x7f8050008650 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:52:25.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.450+0000 7f806e924700 1 -- 172.21.15.90:0/105699411 >> 172.21.15.90:0/105699411 conn(0x7f8068072960 msgr2=0x7f8068072d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:52:25.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.450+0000 7f806e924700 1 -- 172.21.15.90:0/105699411 shutdown_connections 2024-02-09T21:52:25.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.450+0000 7f806e924700 1 -- 172.21.15.90:0/105699411 wait complete. 2024-02-09T21:52:25.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:24 smithi090 ceph-mon[36996]: mgrmap e18: smithi006.glirxt(active, since 27s), standbys: smithi090.csxosj 2024-02-09T21:52:25.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:24 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:25.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:24 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mgr metadata", "who": "smithi090.csxosj", "id": "smithi090.csxosj"}]: dispatch 2024-02-09T21:52:25.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:24 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:52:25.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:24 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-02-09T21:52:25.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.451+0000 7f806e924700 1 Processor -- start 2024-02-09T21:52:25.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.452+0000 7f806e924700 1 -- start start 2024-02-09T21:52:25.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.452+0000 7f806e924700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80500061f0 0x7f80681131c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:52:25.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.452+0000 7f806e924700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8068113700 0x7f806810d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:52:25.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.452+0000 7f806e924700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8068113d10 con 0x7f8068113700 2024-02-09T21:52:25.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.452+0000 7f806e924700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806810d5a0 con 0x7f80500061f0 2024-02-09T21:52:25.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.452+0000 7f8067fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80500061f0 0x7f80681131c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:52:25.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.452+0000 7f8067fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80500061f0 0x7f80681131c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39582/0 (socket says 172.21.15.90:39582) 2024-02-09T21:52:25.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.452+0000 7f8067fff700 1 -- 172.21.15.90:0/4009118073 learned_addr learned my addr 172.21.15.90:0/4009118073 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-09T21:52:25.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.452+0000 7f80677fe700 1 --2- 172.21.15.90:0/4009118073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8068113700 0x7f806810d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:52:25.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.452+0000 7f8067fff700 1 -- 172.21.15.90:0/4009118073 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80500061f0 msgr2=0x7f80681131c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-02-09T21:52:25.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.452+0000 7f8067fff700 1 -- 172.21.15.90:0/4009118073 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80500061f0 msgr2=0x7f80681131c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-02-09T21:52:25.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.452+0000 7f8067fff700 1 --2- 172.21.15.90:0/4009118073 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80500061f0 0x7f80681131c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-02-09T21:52:25.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.452+0000 7f8067fff700 1 --2- 172.21.15.90:0/4009118073 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80500061f0 0x7f80681131c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-09T21:52:25.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.452+0000 7f80677fe700 1 -- 172.21.15.90:0/4009118073 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80500061f0 msgr2=0x7f80681131c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-09T21:52:25.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.452+0000 7f80677fe700 1 --2- 172.21.15.90:0/4009118073 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80500061f0 0x7f80681131c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:52:25.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.452+0000 7f80677fe700 1 -- 172.21.15.90:0/4009118073 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8058005040 con 0x7f8068113700 2024-02-09T21:52:25.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.453+0000 7f80677fe700 1 --2- 172.21.15.90:0/4009118073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8068113700 0x7f806810d060 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f805c00b790 tx=0x7f805c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:52:25.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.453+0000 7f8064ff9700 1 -- 172.21.15.90:0/4009118073 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f805c011810 con 0x7f8068113700 2024-02-09T21:52:25.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.453+0000 7f8064ff9700 1 -- 172.21.15.90:0/4009118073 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f805c011e40 con 0x7f8068113700 2024-02-09T21:52:25.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.453+0000 7f806e924700 1 -- 172.21.15.90:0/4009118073 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f806810d880 con 0x7f8068113700 2024-02-09T21:52:25.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.453+0000 7f8064ff9700 1 -- 172.21.15.90:0/4009118073 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f805c00f2e0 con 0x7f8068113700 2024-02-09T21:52:25.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.453+0000 7f806e924700 1 -- 172.21.15.90:0/4009118073 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f806810ddc0 con 0x7f8068113700 2024-02-09T21:52:25.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.454+0000 7f806e924700 1 -- 172.21.15.90:0/4009118073 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f806805f5c0 con 0x7f8068113700 2024-02-09T21:52:25.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.454+0000 7f8064ff9700 1 -- 172.21.15.90:0/4009118073 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 18) v1 ==== 86563+0+0 (secure 0 0 0) 0x7f805c011970 con 0x7f8068113700 2024-02-09T21:52:25.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.455+0000 7f8064ff9700 1 --2- 172.21.15.90:0/4009118073 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f8054069360 0x7f805406b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:52:25.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.455+0000 7f8064ff9700 1 -- 172.21.15.90:0/4009118073 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f805c0863b0 con 0x7f8068113700 2024-02-09T21:52:25.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.455+0000 7f8067fff700 1 --2- 172.21.15.90:0/4009118073 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f8054069360 0x7f805406b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:52:25.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.455+0000 7f8067fff700 1 --2- 172.21.15.90:0/4009118073 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f8054069360 0x7f805406b810 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8058009ec0 tx=0x7f8058002040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:52:25.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.460+0000 7f8064ff9700 1 -- 172.21.15.90:0/4009118073 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f805c0571d0 con 0x7f8068113700 2024-02-09T21:52:25.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.724+0000 7f806e924700 1 -- 172.21.15.90:0/4009118073 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f806810e7a0 con 0x7f8068113700 2024-02-09T21:52:25.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.725+0000 7f8064ff9700 1 -- 172.21.15.90:0/4009118073 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1009 (secure 0 0 0) 0x7f805c015020 con 0x7f8068113700 2024-02-09T21:52:25.728 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-09T21:52:25.729 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":2,"fsid":"ee26d820-c794-11ee-95b8-87774f69a715","modified":"2024-02-09T21:52:18.874895Z","created":"2024-02-09T21:49:36.342734Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi006","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3300","nonce":0},{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-09T21:52:25.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.727+0000 7f806e924700 1 -- 172.21.15.90:0/4009118073 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f8054069360 msgr2=0x7f805406b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:52:25.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.727+0000 7f806e924700 1 --2- 172.21.15.90:0/4009118073 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f8054069360 0x7f805406b810 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8058009ec0 tx=0x7f8058002040 comp rx=0 tx=0).stop 2024-02-09T21:52:25.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.727+0000 7f806e924700 1 -- 172.21.15.90:0/4009118073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8068113700 msgr2=0x7f806810d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:52:25.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.727+0000 7f806e924700 1 --2- 172.21.15.90:0/4009118073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8068113700 0x7f806810d060 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f805c00b790 tx=0x7f805c0091a0 comp rx=0 tx=0).stop 2024-02-09T21:52:25.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.727+0000 7f806e924700 1 -- 172.21.15.90:0/4009118073 shutdown_connections 2024-02-09T21:52:25.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.727+0000 7f806e924700 1 --2- 172.21.15.90:0/4009118073 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f8054069360 0x7f805406b810 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:52:25.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.727+0000 7f806e924700 1 --2- 172.21.15.90:0/4009118073 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80500061f0 0x7f80681131c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:52:25.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.727+0000 7f806e924700 1 --2- 172.21.15.90:0/4009118073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8068113700 0x7f806810d060 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:52:25.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.727+0000 7f806e924700 1 -- 172.21.15.90:0/4009118073 >> 172.21.15.90:0/4009118073 conn(0x7f8068072960 msgr2=0x7f8068065620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:52:25.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.727+0000 7f806e924700 1 -- 172.21.15.90:0/4009118073 shutdown_connections 2024-02-09T21:52:25.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:52:25.727+0000 7f806e924700 1 -- 172.21.15.90:0/4009118073 wait complete. 2024-02-09T21:52:25.732 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 2 2024-02-09T21:52:26.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:25 smithi006 ceph-mon[29258]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:26.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:25 smithi006 ceph-mon[29258]: from='client.? 172.21.15.90:0/4009118073' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T21:52:26.454 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:25 smithi090 ceph-mon[36996]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:26.454 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:25 smithi090 ceph-mon[36996]: from='client.? 172.21.15.90:0/4009118073' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T21:52:26.533 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-09T21:52:26.533 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph config generate-minimal-conf 2024-02-09T21:52:26.812 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:52:27.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:27 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:27.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:27 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:27.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:27 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:52:27.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:27 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:52:27.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:27 smithi006 ceph-mon[29258]: Updating smithi006:/etc/ceph/ceph.conf 2024-02-09T21:52:27.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:27 smithi006 ceph-mon[29258]: Updating smithi090:/etc/ceph/ceph.conf 2024-02-09T21:52:27.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:27 smithi006 ceph-mon[29258]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:27.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:27 smithi006 ceph-mon[29258]: Updating smithi090:/var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T21:52:27.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:27 smithi006 ceph-mon[29258]: Updating smithi006:/var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T21:52:27.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:27 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:52:27.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:27 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:27.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:27 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:27.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:27 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:27.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:27 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:27.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:27 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:27.909 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:27 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:27.909 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:27 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T21:52:27.909 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:27 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-09T21:52:27.909 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:27 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:52:27.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:27 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:27.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:27 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:27.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:27 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:52:27.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:27 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:52:27.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:27 smithi090 ceph-mon[36996]: Updating smithi006:/etc/ceph/ceph.conf 2024-02-09T21:52:27.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:27 smithi090 ceph-mon[36996]: Updating smithi090:/etc/ceph/ceph.conf 2024-02-09T21:52:27.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:27 smithi090 ceph-mon[36996]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:27.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:27 smithi090 ceph-mon[36996]: Updating smithi090:/var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T21:52:27.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:27 smithi090 ceph-mon[36996]: Updating smithi006:/var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T21:52:27.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:27 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:52:27.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:27 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:27.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:27 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:27.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:27 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:27.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:27 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:27.956 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:27 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:27.956 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:27 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:27.956 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:27 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T21:52:27.956 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:27 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-09T21:52:27.956 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:27 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:52:28.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:28 smithi006 ceph-mon[29258]: Reconfiguring mon.smithi006 (unknown last config time)... 2024-02-09T21:52:28.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:28 smithi006 ceph-mon[29258]: Reconfiguring daemon mon.smithi006 on smithi006 2024-02-09T21:52:28.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:28 smithi090 ceph-mon[36996]: Reconfiguring mon.smithi006 (unknown last config time)... 2024-02-09T21:52:28.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:28 smithi090 ceph-mon[36996]: Reconfiguring daemon mon.smithi006 on smithi006 2024-02-09T21:52:29.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.136+0000 7fb02a72b700 1 -- 172.21.15.6:0/752462025 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb024101b60 msgr2=0x7fb024101f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:52:29.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.136+0000 7fb02a72b700 1 --2- 172.21.15.6:0/752462025 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb024101b60 0x7fb024101f30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb0140048c0 tx=0x7fb014009e30 comp rx=0 tx=0).stop 2024-02-09T21:52:29.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.136+0000 7fb02a72b700 1 -- 172.21.15.6:0/752462025 shutdown_connections 2024-02-09T21:52:29.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.136+0000 7fb02a72b700 1 --2- 172.21.15.6:0/752462025 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb024101b60 0x7fb024101f30 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:52:29.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.136+0000 7fb02a72b700 1 -- 172.21.15.6:0/752462025 >> 172.21.15.6:0/752462025 conn(0x7fb024072850 msgr2=0x7fb024072c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:52:29.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.136+0000 7fb02a72b700 1 -- 172.21.15.6:0/752462025 shutdown_connections 2024-02-09T21:52:29.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.136+0000 7fb02a72b700 1 -- 172.21.15.6:0/752462025 wait complete. 2024-02-09T21:52:29.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.136+0000 7fb02a72b700 1 Processor -- start 2024-02-09T21:52:29.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.137+0000 7fb02a72b700 1 -- start start 2024-02-09T21:52:29.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.137+0000 7fb02a72b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb024101b60 0x7fb02410e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:52:29.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.137+0000 7fb02a72b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb02410e650 0x7fb024116fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:52:29.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.137+0000 7fb02a72b700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02410eb50 con 0x7fb024101b60 2024-02-09T21:52:29.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.137+0000 7fb02a72b700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02410ecc0 con 0x7fb02410e650 2024-02-09T21:52:29.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.137+0000 7fb023fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb024101b60 0x7fb02410e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:52:29.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.137+0000 7fb023fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb024101b60 0x7fb02410e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36818/0 (socket says 172.21.15.6:36818) 2024-02-09T21:52:29.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.137+0000 7fb023fff700 1 -- 172.21.15.6:0/2708358356 learned_addr learned my addr 172.21.15.6:0/2708358356 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:52:29.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.137+0000 7fb023fff700 1 -- 172.21.15.6:0/2708358356 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb02410e650 msgr2=0x7fb024116fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:52:29.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.137+0000 7fb023fff700 1 --2- 172.21.15.6:0/2708358356 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb02410e650 0x7fb024116fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:52:29.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.137+0000 7fb023fff700 1 -- 172.21.15.6:0/2708358356 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb014005040 con 0x7fb024101b60 2024-02-09T21:52:29.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.138+0000 7fb023fff700 1 --2- 172.21.15.6:0/2708358356 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb024101b60 0x7fb02410e110 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb014000c00 tx=0x7fb014009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:52:29.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.138+0000 7fb020ff9700 1 -- 172.21.15.6:0/2708358356 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb01400e040 con 0x7fb024101b60 2024-02-09T21:52:29.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.138+0000 7fb020ff9700 1 -- 172.21.15.6:0/2708358356 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb014009760 con 0x7fb024101b60 2024-02-09T21:52:29.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.138+0000 7fb02a72b700 1 -- 172.21.15.6:0/2708358356 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb024117500 con 0x7fb024101b60 2024-02-09T21:52:29.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.138+0000 7fb020ff9700 1 -- 172.21.15.6:0/2708358356 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb01401c630 con 0x7fb024101b60 2024-02-09T21:52:29.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.138+0000 7fb02a72b700 1 -- 172.21.15.6:0/2708358356 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb024117a90 con 0x7fb024101b60 2024-02-09T21:52:29.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.139+0000 7fb02a72b700 1 -- 172.21.15.6:0/2708358356 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb02405f5c0 con 0x7fb024101b60 2024-02-09T21:52:29.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.139+0000 7fb020ff9700 1 -- 172.21.15.6:0/2708358356 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 18) v1 ==== 86563+0+0 (secure 0 0 0) 0x7fb014013420 con 0x7fb024101b60 2024-02-09T21:52:29.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.140+0000 7fb020ff9700 1 --2- 172.21.15.6:0/2708358356 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fb00c0693d0 0x7fb00c06b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:52:29.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.140+0000 7fb020ff9700 1 -- 172.21.15.6:0/2708358356 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fb014086f30 con 0x7fb024101b60 2024-02-09T21:52:29.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.140+0000 7fb0237fe700 1 --2- 172.21.15.6:0/2708358356 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fb00c0693d0 0x7fb00c06b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:52:29.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.140+0000 7fb0237fe700 1 --2- 172.21.15.6:0/2708358356 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fb00c0693d0 0x7fb00c06b880 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb018009930 tx=0x7fb018006de0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:52:29.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.144+0000 7fb020ff9700 1 -- 172.21.15.6:0/2708358356 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb014054350 con 0x7fb024101b60 2024-02-09T21:52:29.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.340+0000 7fb02a72b700 1 -- 172.21.15.6:0/2708358356 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fb024065960 con 0x7fb024101b60 2024-02-09T21:52:29.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.341+0000 7fb020ff9700 1 -- 172.21.15.6:0/2708358356 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+221 (secure 0 0 0) 0x7fb0140210f0 con 0x7fb024101b60 2024-02-09T21:52:29.344 INFO:teuthology.orchestra.run.smithi006.stdout:# minimal ceph.conf for ee26d820-c794-11ee-95b8-87774f69a715 2024-02-09T21:52:29.344 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-02-09T21:52:29.345 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = ee26d820-c794-11ee-95b8-87774f69a715 2024-02-09T21:52:29.345 INFO:teuthology.orchestra.run.smithi006.stdout: mon_host = [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2024-02-09T21:52:29.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.342+0000 7fb02a72b700 1 -- 172.21.15.6:0/2708358356 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fb00c0693d0 msgr2=0x7fb00c06b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:52:29.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.342+0000 7fb02a72b700 1 --2- 172.21.15.6:0/2708358356 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fb00c0693d0 0x7fb00c06b880 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb018009930 tx=0x7fb018006de0 comp rx=0 tx=0).stop 2024-02-09T21:52:29.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.342+0000 7fb02a72b700 1 -- 172.21.15.6:0/2708358356 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb024101b60 msgr2=0x7fb02410e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:52:29.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.342+0000 7fb02a72b700 1 --2- 172.21.15.6:0/2708358356 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb024101b60 0x7fb02410e110 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb014000c00 tx=0x7fb014009e30 comp rx=0 tx=0).stop 2024-02-09T21:52:29.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.343+0000 7fb02a72b700 1 -- 172.21.15.6:0/2708358356 shutdown_connections 2024-02-09T21:52:29.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.343+0000 7fb02a72b700 1 --2- 172.21.15.6:0/2708358356 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fb00c0693d0 0x7fb00c06b880 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:52:29.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.343+0000 7fb02a72b700 1 --2- 172.21.15.6:0/2708358356 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb024101b60 0x7fb02410e110 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:52:29.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.343+0000 7fb02a72b700 1 --2- 172.21.15.6:0/2708358356 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb02410e650 0x7fb024116fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:52:29.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.343+0000 7fb02a72b700 1 -- 172.21.15.6:0/2708358356 >> 172.21.15.6:0/2708358356 conn(0x7fb024072850 msgr2=0x7fb0240fd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:52:29.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.343+0000 7fb02a72b700 1 -- 172.21.15.6:0/2708358356 shutdown_connections 2024-02-09T21:52:29.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:29.343+0000 7fb02a72b700 1 -- 172.21.15.6:0/2708358356 wait complete. 2024-02-09T21:52:30.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:29 smithi006 ceph-mon[29258]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:30.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:29 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/2708358356' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:52:30.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:29 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:30.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:29 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:30.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:29 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi006.glirxt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-09T21:52:30.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:29 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-09T21:52:30.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:29 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:52:30.186 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-09T21:52:30.186 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-02-09T21:52:30.186 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/ceph.conf 2024-02-09T21:52:30.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:29 smithi090 ceph-mon[36996]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:30.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:29 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/2708358356' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:52:30.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:29 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:30.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:29 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:30.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:29 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi006.glirxt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-09T21:52:30.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:29 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-09T21:52:30.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:29 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:52:30.218 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-02-09T21:52:30.218 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-09T21:52:30.286 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-09T21:52:30.286 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.conf 2024-02-09T21:52:30.317 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-09T21:52:30.317 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-09T21:52:30.385 INFO:tasks.cephadm:Deploying OSDs... 2024-02-09T21:52:30.385 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-02-09T21:52:30.385 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/scratch_devs of=/dev/stdout 2024-02-09T21:52:30.400 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-09T21:52:30.400 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_1 2024-02-09T21:52:30.455 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-09T21:52:30.455 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T21:52:30.455 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-02-09T21:52:30.455 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T21:52:30.455 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T21:52:30.455 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-02-09 21:51:48.444473683 +0000 2024-02-09T21:52:30.455 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-02-09 21:45:32.531153870 +0000 2024-02-09T21:52:30.455 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-02-09 21:45:32.531153870 +0000 2024-02-09T21:52:30.455 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-02-09 21:45:32.531153870 +0000 2024-02-09T21:52:30.456 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-09T21:52:30.521 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-02-09T21:52:30.522 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-02-09T21:52:30.522 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000199978 s, 2.6 MB/s 2024-02-09T21:52:30.523 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-09T21:52:30.579 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_2 2024-02-09T21:52:30.634 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-09T21:52:30.634 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T21:52:30.634 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-02-09T21:52:30.634 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T21:52:30.634 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T21:52:30.635 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-02-09 21:51:48.444473683 +0000 2024-02-09T21:52:30.635 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-02-09 21:45:32.890149886 +0000 2024-02-09T21:52:30.635 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-02-09 21:45:32.890149886 +0000 2024-02-09T21:52:30.635 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-02-09 21:45:32.890149886 +0000 2024-02-09T21:52:30.635 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-09T21:52:30.700 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-02-09T21:52:30.700 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-02-09T21:52:30.701 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000197438 s, 2.6 MB/s 2024-02-09T21:52:30.702 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-09T21:52:30.757 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_3 2024-02-09T21:52:30.811 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-09T21:52:30.812 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T21:52:30.812 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-02-09T21:52:30.812 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T21:52:30.812 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T21:52:30.812 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-02-09 21:51:48.444473683 +0000 2024-02-09T21:52:30.812 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-02-09 21:45:33.237146035 +0000 2024-02-09T21:52:30.812 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-02-09 21:45:33.237146035 +0000 2024-02-09T21:52:30.812 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-02-09 21:45:33.237146035 +0000 2024-02-09T21:52:30.812 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-09T21:52:30.879 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-02-09T21:52:30.879 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-02-09T21:52:30.879 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000194483 s, 2.6 MB/s 2024-02-09T21:52:30.880 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-09T21:52:30.935 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_4 2024-02-09T21:52:30.989 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-09T21:52:30.990 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T21:52:30.990 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-02-09T21:52:30.990 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T21:52:30.990 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T21:52:30.990 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-02-09 21:51:48.445473672 +0000 2024-02-09T21:52:30.990 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-02-09 21:45:33.597142040 +0000 2024-02-09T21:52:30.990 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-02-09 21:45:33.597142040 +0000 2024-02-09T21:52:30.990 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-02-09 21:45:33.597142040 +0000 2024-02-09T21:52:30.990 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-09T21:52:31.053 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:30 smithi006 ceph-mon[29258]: Reconfiguring mgr.smithi006.glirxt (unknown last config time)... 2024-02-09T21:52:31.053 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:30 smithi006 ceph-mon[29258]: Reconfiguring daemon mgr.smithi006.glirxt on smithi006 2024-02-09T21:52:31.056 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-02-09T21:52:31.056 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-02-09T21:52:31.056 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000184328 s, 2.8 MB/s 2024-02-09T21:52:31.057 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-09T21:52:31.112 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-09T21:52:31.112 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/scratch_devs of=/dev/stdout 2024-02-09T21:52:31.128 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-09T21:52:31.128 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_1 2024-02-09T21:52:31.183 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-09T21:52:31.183 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T21:52:31.183 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-02-09T21:52:31.183 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T21:52:31.183 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T21:52:31.183 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-02-09 21:52:17.539670419 +0000 2024-02-09T21:52:31.183 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-02-09 21:45:32.531058963 +0000 2024-02-09T21:52:31.183 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-02-09 21:45:32.531058963 +0000 2024-02-09T21:52:31.183 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-02-09 21:45:32.531058963 +0000 2024-02-09T21:52:31.184 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-09T21:52:31.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:30 smithi090 ceph-mon[36996]: Reconfiguring mgr.smithi006.glirxt (unknown last config time)... 2024-02-09T21:52:31.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:30 smithi090 ceph-mon[36996]: Reconfiguring daemon mgr.smithi006.glirxt on smithi006 2024-02-09T21:52:31.229 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-02-09T21:52:31.229 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-02-09T21:52:31.229 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000247747 s, 2.1 MB/s 2024-02-09T21:52:31.230 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-09T21:52:31.285 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_2 2024-02-09T21:52:31.340 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-09T21:52:31.340 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T21:52:31.340 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-02-09T21:52:31.340 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T21:52:31.340 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T21:52:31.340 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-02-09 21:52:17.539670419 +0000 2024-02-09T21:52:31.341 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-02-09 21:45:32.866053194 +0000 2024-02-09T21:52:31.341 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-02-09 21:45:32.866053194 +0000 2024-02-09T21:52:31.341 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-02-09 21:45:32.866053194 +0000 2024-02-09T21:52:31.341 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-09T21:52:31.408 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-02-09T21:52:31.408 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-02-09T21:52:31.408 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000175836 s, 2.9 MB/s 2024-02-09T21:52:31.410 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-09T21:52:31.466 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_3 2024-02-09T21:52:31.521 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-09T21:52:31.521 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T21:52:31.521 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-02-09T21:52:31.521 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T21:52:31.521 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T21:52:31.521 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-02-09 21:52:17.539670419 +0000 2024-02-09T21:52:31.521 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-02-09 21:45:33.202047407 +0000 2024-02-09T21:52:31.521 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-02-09 21:45:33.202047407 +0000 2024-02-09T21:52:31.522 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-02-09 21:45:33.202047407 +0000 2024-02-09T21:52:31.522 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-09T21:52:31.587 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-02-09T21:52:31.587 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-02-09T21:52:31.587 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000272324 s, 1.9 MB/s 2024-02-09T21:52:31.588 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-09T21:52:31.643 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_4 2024-02-09T21:52:31.698 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-09T21:52:31.698 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T21:52:31.698 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-02-09T21:52:31.698 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T21:52:31.698 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T21:52:31.698 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-02-09 21:52:17.540670402 +0000 2024-02-09T21:52:31.698 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-02-09 21:45:33.551041396 +0000 2024-02-09T21:52:31.698 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-02-09 21:45:33.551041396 +0000 2024-02-09T21:52:31.698 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-02-09 21:45:33.551041396 +0000 2024-02-09T21:52:31.698 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-09T21:52:31.763 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-02-09T21:52:31.763 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-02-09T21:52:31.763 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.0001788 s, 2.9 MB/s 2024-02-09T21:52:31.764 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-09T21:52:31.820 INFO:tasks.cephadm:Deploying osd.0 on smithi006 with /dev/vg_nvme/lv_4... 2024-02-09T21:52:31.820 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-09T21:52:32.116 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:31 smithi006 ceph-mon[29258]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:32.116 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:31 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:32.116 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:31 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:32.116 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:31 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi006", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-09T21:52:32.116 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:31 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:52:32.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:31 smithi090 ceph-mon[36996]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:32.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:31 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:32.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:31 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:32.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:31 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi006", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-09T21:52:32.206 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:31 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:52:32.776 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:52:33.130 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:32 smithi006 ceph-mon[29258]: Reconfiguring crash.smithi006 (monmap changed)... 2024-02-09T21:52:33.130 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:32 smithi006 ceph-mon[29258]: Reconfiguring daemon crash.smithi006 on smithi006 2024-02-09T21:52:33.131 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:32 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:33.131 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:32 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:33.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:32 smithi090 ceph-mon[36996]: Reconfiguring crash.smithi006 (monmap changed)... 2024-02-09T21:52:33.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:32 smithi090 ceph-mon[36996]: Reconfiguring daemon crash.smithi006 on smithi006 2024-02-09T21:52:33.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:32 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:33.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:32 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:34.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:33 smithi006 ceph-mon[29258]: Reconfiguring alertmanager.smithi006 (dependencies changed)... 2024-02-09T21:52:34.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:33 smithi006 ceph-mon[29258]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:34.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:33 smithi006 ceph-mon[29258]: Reconfiguring daemon alertmanager.smithi006 on smithi006 2024-02-09T21:52:34.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:33 smithi090 ceph-mon[36996]: Reconfiguring alertmanager.smithi006 (dependencies changed)... 2024-02-09T21:52:34.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:33 smithi090 ceph-mon[36996]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:34.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:33 smithi090 ceph-mon[36996]: Reconfiguring daemon alertmanager.smithi006 on smithi006 2024-02-09T21:52:36.137 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:35 smithi006 ceph-mon[29258]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:36.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:35 smithi090 ceph-mon[36996]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:36.705 INFO:teuthology.orchestra.run.smithi006.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-09T21:52:36.705 INFO:teuthology.orchestra.run.smithi006.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-09T21:52:36.705 INFO:teuthology.orchestra.run.smithi006.stderr: stderr: 10+0 records in 2024-02-09T21:52:36.706 INFO:teuthology.orchestra.run.smithi006.stderr:10+0 records out 2024-02-09T21:52:36.706 INFO:teuthology.orchestra.run.smithi006.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124467 s, 842 MB/s 2024-02-09T21:52:36.706 INFO:teuthology.orchestra.run.smithi006.stderr:--> Zapping successful for: 2024-02-09T21:52:37.498 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:37 smithi006 ceph-mon[29258]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:37.521 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi006:vg_nvme/lv_4 2024-02-09T21:52:37.704 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:37 smithi090 ceph-mon[36996]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:38.512 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:52:40.135 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:39 smithi006 ceph-mon[29258]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:40.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:39 smithi090 ceph-mon[36996]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:40.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.123+0000 7f11bce57700 1 -- 172.21.15.6:0/596898737 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11b80fcef0 msgr2=0x7f11b80fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:52:40.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.123+0000 7f11bce57700 1 --2- 172.21.15.6:0/596898737 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11b80fcef0 0x7f11b80fd320 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f11a8004940 tx=0x7f11a8009e30 comp rx=0 tx=0).stop 2024-02-09T21:52:40.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.123+0000 7f11bce57700 1 -- 172.21.15.6:0/596898737 shutdown_connections 2024-02-09T21:52:40.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.123+0000 7f11bce57700 1 --2- 172.21.15.6:0/596898737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11b80fe0e0 0x7f11b80fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:52:40.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.123+0000 7f11bce57700 1 --2- 172.21.15.6:0/596898737 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11b80fcef0 0x7f11b80fd320 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:52:40.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.123+0000 7f11bce57700 1 -- 172.21.15.6:0/596898737 >> 172.21.15.6:0/596898737 conn(0x7f11b80f8620 msgr2=0x7f11b80faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:52:40.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.123+0000 7f11bce57700 1 -- 172.21.15.6:0/596898737 shutdown_connections 2024-02-09T21:52:40.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.123+0000 7f11bce57700 1 -- 172.21.15.6:0/596898737 wait complete. 2024-02-09T21:52:40.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.124+0000 7f11bce57700 1 Processor -- start 2024-02-09T21:52:40.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.124+0000 7f11bce57700 1 -- start start 2024-02-09T21:52:40.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.124+0000 7f11bce57700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11b80fcef0 0x7f11b810d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:52:40.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.124+0000 7f11bce57700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11b80fe0e0 0x7f11b810deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:52:40.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.124+0000 7f11bce57700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b810e4b0 con 0x7f11b80fcef0 2024-02-09T21:52:40.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.124+0000 7f11bce57700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b810ed50 con 0x7f11b80fe0e0 2024-02-09T21:52:40.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.124+0000 7f11b659c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11b80fcef0 0x7f11b810d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:52:40.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.124+0000 7f11b659c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11b80fcef0 0x7f11b810d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58720/0 (socket says 172.21.15.6:58720) 2024-02-09T21:52:40.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.124+0000 7f11b659c700 1 -- 172.21.15.6:0/2002829540 learned_addr learned my addr 172.21.15.6:0/2002829540 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:52:40.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.124+0000 7f11b659c700 1 -- 172.21.15.6:0/2002829540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11b80fe0e0 msgr2=0x7f11b810deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:52:40.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.124+0000 7f11b659c700 1 --2- 172.21.15.6:0/2002829540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11b80fe0e0 0x7f11b810deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:52:40.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.124+0000 7f11b659c700 1 -- 172.21.15.6:0/2002829540 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a8005040 con 0x7f11b80fcef0 2024-02-09T21:52:40.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.125+0000 7f11b659c700 1 --2- 172.21.15.6:0/2002829540 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11b80fcef0 0x7f11b810d970 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f11a8009360 tx=0x7f11a8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:52:40.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.125+0000 7f11aeffd700 1 -- 172.21.15.6:0/2002829540 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11a800e040 con 0x7f11b80fcef0 2024-02-09T21:52:40.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.125+0000 7f11aeffd700 1 -- 172.21.15.6:0/2002829540 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f11a8004290 con 0x7f11b80fcef0 2024-02-09T21:52:40.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.125+0000 7f11bce57700 1 -- 172.21.15.6:0/2002829540 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11b810efd0 con 0x7f11b80fcef0 2024-02-09T21:52:40.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.125+0000 7f11aeffd700 1 -- 172.21.15.6:0/2002829540 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11a801c630 con 0x7f11b80fcef0 2024-02-09T21:52:40.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.125+0000 7f11bce57700 1 -- 172.21.15.6:0/2002829540 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11b810f510 con 0x7f11b80fcef0 2024-02-09T21:52:40.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.126+0000 7f11bce57700 1 -- 172.21.15.6:0/2002829540 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11b804bed0 con 0x7f11b80fcef0 2024-02-09T21:52:40.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.126+0000 7f11aeffd700 1 -- 172.21.15.6:0/2002829540 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 18) v1 ==== 86563+0+0 (secure 0 0 0) 0x7f11a800bcc0 con 0x7f11b80fcef0 2024-02-09T21:52:40.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.126+0000 7f11aeffd700 1 --2- 172.21.15.6:0/2002829540 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f11a4069210 0x7f11a406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:52:40.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.126+0000 7f11aeffd700 1 -- 172.21.15.6:0/2002829540 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f11a80874a0 con 0x7f11b80fcef0 2024-02-09T21:52:40.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.127+0000 7f11b5d9b700 1 --2- 172.21.15.6:0/2002829540 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f11a4069210 0x7f11a406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:52:40.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.127+0000 7f11b5d9b700 1 --2- 172.21.15.6:0/2002829540 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f11a4069210 0x7f11a406b6c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f11a0004b70 tx=0x7f11a0007cc0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:52:40.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.131+0000 7f11aeffd700 1 -- 172.21.15.6:0/2002829540 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f11a8058340 con 0x7f11b80fcef0 2024-02-09T21:52:40.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:52:40.332+0000 7f11bce57700 1 -- 172.21.15.6:0/2002829540 --> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f11b8102ac0 con 0x7f11a4069210 2024-02-09T21:52:41.107 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:40 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T21:52:41.108 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:40 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T21:52:41.108 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:40 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:52:41.108 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:40 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:41.108 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:40 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:41.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:40 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T21:52:41.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:40 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T21:52:41.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:40 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:52:41.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:40 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:41.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:40 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:42.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:41 smithi006 ceph-mon[29258]: from='client.14248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:52:42.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:41 smithi006 ceph-mon[29258]: Reconfiguring grafana.smithi006 (dependencies changed)... 2024-02-09T21:52:42.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:41 smithi006 ceph-mon[29258]: Reconfiguring daemon grafana.smithi006 on smithi006 2024-02-09T21:52:42.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:41 smithi006 ceph-mon[29258]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:42.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:41 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:52:42.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:41 smithi090 ceph-mon[36996]: from='client.14248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:52:42.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:41 smithi090 ceph-mon[36996]: Reconfiguring grafana.smithi006 (dependencies changed)... 2024-02-09T21:52:42.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:41 smithi090 ceph-mon[36996]: Reconfiguring daemon grafana.smithi006 on smithi006 2024-02-09T21:52:42.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:41 smithi090 ceph-mon[36996]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:42.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:41 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:52:44.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:43 smithi006 ceph-mon[29258]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:44.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:43 smithi090 ceph-mon[36996]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:46.046 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:45 smithi006 ceph-mon[29258]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:46.046 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:45 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:46.046 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:45 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:46.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:45 smithi090 ceph-mon[36996]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:46.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:45 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:46.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:45 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:46.943 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:46 smithi006 ceph-mon[29258]: Reconfiguring prometheus.smithi006 (dependencies changed)... 2024-02-09T21:52:46.943 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:46 smithi006 ceph-mon[29258]: Reconfiguring daemon prometheus.smithi006 on smithi006 2024-02-09T21:52:47.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:46 smithi090 ceph-mon[36996]: Reconfiguring prometheus.smithi006 (dependencies changed)... 2024-02-09T21:52:47.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:46 smithi090 ceph-mon[36996]: Reconfiguring daemon prometheus.smithi006 on smithi006 2024-02-09T21:52:48.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:47 smithi090 ceph-mon[36996]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:48.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:47 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/3831026743' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c78f3c5b-14ab-40b7-bd39-52cb040b6532"}]: dispatch 2024-02-09T21:52:48.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:47 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/3831026743' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c78f3c5b-14ab-40b7-bd39-52cb040b6532"}]': finished 2024-02-09T21:52:48.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:47 smithi090 ceph-mon[36996]: osdmap e6: 1 total, 0 up, 1 in 2024-02-09T21:52:48.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:47 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T21:52:48.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:47 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/4102698720' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T21:52:48.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:47 smithi006 ceph-mon[29258]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:48.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:47 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/3831026743' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c78f3c5b-14ab-40b7-bd39-52cb040b6532"}]: dispatch 2024-02-09T21:52:48.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:47 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/3831026743' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c78f3c5b-14ab-40b7-bd39-52cb040b6532"}]': finished 2024-02-09T21:52:48.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:47 smithi006 ceph-mon[29258]: osdmap e6: 1 total, 0 up, 1 in 2024-02-09T21:52:48.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:47 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T21:52:48.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:47 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/4102698720' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T21:52:50.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:49 smithi090 ceph-mon[36996]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:50.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:49 smithi006 ceph-mon[29258]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:52.018 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:51 smithi006 ceph-mon[29258]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:52.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:51 smithi090 ceph-mon[36996]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:54.006 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:53 smithi006 ceph-mon[29258]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:54.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:53 smithi090 ceph-mon[36996]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:56.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:55 smithi090 ceph-mon[36996]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:56.250 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:55 smithi006 ceph-mon[29258]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:57.620 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:57 smithi090 ceph-mon[36996]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:57.620 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:57 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:52:57.620 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:57 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:57.620 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:57 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:57.620 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:57 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi090", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-09T21:52:57.620 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:57 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:52:57.634 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:57 smithi006 ceph-mon[29258]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:57.634 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:57 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:52:57.634 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:57 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:57.634 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:57 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:57.634 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:57 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi090", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-09T21:52:57.634 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:57 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:52:58.562 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:58 smithi006 ceph-mon[29258]: Reconfiguring crash.smithi090 (monmap changed)... 2024-02-09T21:52:58.562 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:58 smithi006 ceph-mon[29258]: Reconfiguring daemon crash.smithi090 on smithi090 2024-02-09T21:52:58.625 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:58 smithi090 ceph-mon[36996]: Reconfiguring crash.smithi090 (monmap changed)... 2024-02-09T21:52:58.625 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:58 smithi090 ceph-mon[36996]: Reconfiguring daemon crash.smithi090 on smithi090 2024-02-09T21:52:59.836 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:59 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:59.836 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:59 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:59.836 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:59 smithi006 ceph-mon[29258]: Reconfiguring mgr.smithi090.csxosj (monmap changed)... 2024-02-09T21:52:59.836 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:59 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi090.csxosj", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-09T21:52:59.836 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:59 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-09T21:52:59.836 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:59 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:52:59.836 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:59 smithi006 ceph-mon[29258]: Reconfiguring daemon mgr.smithi090.csxosj on smithi090 2024-02-09T21:52:59.837 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:59 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-09T21:52:59.837 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:59 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:52:59.837 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:59 smithi006 ceph-mon[29258]: Deploying daemon osd.0 on smithi006 2024-02-09T21:52:59.837 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:52:59 smithi006 ceph-mon[29258]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:52:59.871 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:59 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:59.871 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:59 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:52:59.871 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:59 smithi090 ceph-mon[36996]: Reconfiguring mgr.smithi090.csxosj (monmap changed)... 2024-02-09T21:52:59.872 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:59 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi090.csxosj", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-09T21:52:59.872 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:59 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-09T21:52:59.872 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:59 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:52:59.872 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:59 smithi090 ceph-mon[36996]: Reconfiguring daemon mgr.smithi090.csxosj on smithi090 2024-02-09T21:52:59.872 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:59 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-09T21:52:59.872 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:59 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:52:59.872 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:59 smithi090 ceph-mon[36996]: Deploying daemon osd.0 on smithi006 2024-02-09T21:52:59.872 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:52:59 smithi090 ceph-mon[36996]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:53:01.192 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:00 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:01.192 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:00 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:01.192 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:00 smithi006 ceph-mon[29258]: Reconfiguring mon.smithi090 (monmap changed)... 2024-02-09T21:53:01.192 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:00 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T21:53:01.192 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:00 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-09T21:53:01.192 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:00 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:53:01.192 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:00 smithi006 ceph-mon[29258]: Reconfiguring daemon mon.smithi090 on smithi090 2024-02-09T21:53:01.196 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:00 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:01.196 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:00 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:01.196 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:00 smithi090 ceph-mon[36996]: Reconfiguring mon.smithi090 (monmap changed)... 2024-02-09T21:53:01.196 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:00 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T21:53:01.196 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:00 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-09T21:53:01.196 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:00 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:53:01.196 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:00 smithi090 ceph-mon[36996]: Reconfiguring daemon mon.smithi090 on smithi090 2024-02-09T21:53:02.125 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:01 smithi006 ceph-mon[29258]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:53:02.125 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:02.125 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:02.126 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-09T21:53:02.126 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi006.front.sepia.ceph.com:9093"}]: dispatch 2024-02-09T21:53:02.126 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:02.126 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-09T21:53:02.126 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi006.front.sepia.ceph.com:3000"}]: dispatch 2024-02-09T21:53:02.126 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:02.126 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-09T21:53:02.126 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi006.front.sepia.ceph.com:9095"}]: dispatch 2024-02-09T21:53:02.126 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:02.126 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:53:02.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:01 smithi090 ceph-mon[36996]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:53:02.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:02.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:02.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-09T21:53:02.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi006.front.sepia.ceph.com:9093"}]: dispatch 2024-02-09T21:53:02.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:02.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-09T21:53:02.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi006.front.sepia.ceph.com:3000"}]: dispatch 2024-02-09T21:53:02.206 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:02.206 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-09T21:53:02.206 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi006.front.sepia.ceph.com:9095"}]: dispatch 2024-02-09T21:53:02.206 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:02.206 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:53:03.184 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:02 smithi006 ceph-mon[29258]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-09T21:53:03.184 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:02 smithi006 ceph-mon[29258]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi006.front.sepia.ceph.com:9093"}]: dispatch 2024-02-09T21:53:03.185 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:02 smithi006 ceph-mon[29258]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-09T21:53:03.185 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:02 smithi006 ceph-mon[29258]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi006.front.sepia.ceph.com:3000"}]: dispatch 2024-02-09T21:53:03.185 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:02 smithi006 ceph-mon[29258]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-09T21:53:03.185 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:02 smithi006 ceph-mon[29258]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi006.front.sepia.ceph.com:9095"}]: dispatch 2024-02-09T21:53:03.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:02 smithi090 ceph-mon[36996]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-09T21:53:03.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:02 smithi090 ceph-mon[36996]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi006.front.sepia.ceph.com:9093"}]: dispatch 2024-02-09T21:53:03.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:02 smithi090 ceph-mon[36996]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-09T21:53:03.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:02 smithi090 ceph-mon[36996]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi006.front.sepia.ceph.com:3000"}]: dispatch 2024-02-09T21:53:03.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:02 smithi090 ceph-mon[36996]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-09T21:53:03.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:02 smithi090 ceph-mon[36996]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi006.front.sepia.ceph.com:9095"}]: dispatch 2024-02-09T21:53:04.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:04 smithi090 ceph-mon[36996]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:53:04.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:04 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:04.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:04 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:04.547 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:04 smithi006 ceph-mon[29258]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:53:04.547 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:04 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:04.548 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:04 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:06.296 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:06 smithi006 ceph-mon[29258]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:53:06.296 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:06 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:06.296 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:06 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:06.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:06 smithi090 ceph-mon[36996]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:53:06.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:06 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:06.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:06 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:07.642 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:07 smithi006 ceph-mon[29258]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:53:07.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:07 smithi090 ceph-mon[36996]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:53:08.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:08.857+0000 7f11aeffd700 1 -- 172.21.15.6:0/2002829540 <== mgr.14216 v2:172.21.15.6:6800/2624137529 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f11b8102ac0 con 0x7f11a4069210 2024-02-09T21:53:08.862 INFO:teuthology.orchestra.run.smithi006.stdout:Created osd(s) 0 on host 'smithi006' 2024-02-09T21:53:08.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:08.860+0000 7f11bce57700 1 -- 172.21.15.6:0/2002829540 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f11a4069210 msgr2=0x7f11a406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:53:08.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:08.860+0000 7f11bce57700 1 --2- 172.21.15.6:0/2002829540 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f11a4069210 0x7f11a406b6c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f11a0004b70 tx=0x7f11a0007cc0 comp rx=0 tx=0).stop 2024-02-09T21:53:08.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:08.860+0000 7f11bce57700 1 -- 172.21.15.6:0/2002829540 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11b80fcef0 msgr2=0x7f11b810d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:53:08.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:08.860+0000 7f11bce57700 1 --2- 172.21.15.6:0/2002829540 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11b80fcef0 0x7f11b810d970 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f11a8009360 tx=0x7f11a8009e30 comp rx=0 tx=0).stop 2024-02-09T21:53:08.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:08.861+0000 7f11bce57700 1 -- 172.21.15.6:0/2002829540 shutdown_connections 2024-02-09T21:53:08.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:08.861+0000 7f11bce57700 1 --2- 172.21.15.6:0/2002829540 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f11a4069210 0x7f11a406b6c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:53:08.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:08.861+0000 7f11bce57700 1 --2- 172.21.15.6:0/2002829540 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11b80fcef0 0x7f11b810d970 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:53:08.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:08.861+0000 7f11bce57700 1 --2- 172.21.15.6:0/2002829540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11b80fe0e0 0x7f11b810deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:53:08.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:08.861+0000 7f11bce57700 1 -- 172.21.15.6:0/2002829540 >> 172.21.15.6:0/2002829540 conn(0x7f11b80f8620 msgr2=0x7f11b81013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:53:08.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:08.861+0000 7f11bce57700 1 -- 172.21.15.6:0/2002829540 shutdown_connections 2024-02-09T21:53:08.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:08.861+0000 7f11bce57700 1 -- 172.21.15.6:0/2002829540 wait complete. 2024-02-09T21:53:09.113 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:08 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:09.113 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:08 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:09.113 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:08 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:53:09.113 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:08 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:53:09.113 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:08 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:09.113 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:08 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:53:09.113 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:08 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:09.113 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:08 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:09.283 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:08 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:09.284 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:08 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:09.284 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:08 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:53:09.284 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:08 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:53:09.284 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:08 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:09.284 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:08 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:53:09.284 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:08 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:09.284 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:08 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:09.433 DEBUG:teuthology.orchestra.run.smithi006:osd.0> sudo journalctl -f -n 0 -u ceph-ee26d820-c794-11ee-95b8-87774f69a715@osd.0.service 2024-02-09T21:53:09.436 INFO:tasks.cephadm:Deploying osd.1 on smithi006 with /dev/vg_nvme/lv_3... 2024-02-09T21:53:09.436 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-09T21:53:10.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:09 smithi006 ceph-mon[29258]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:53:10.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:09 smithi006 ceph-mon[29258]: from='osd.0 [v2:172.21.15.6:6802/4292395427,v1:172.21.15.6:6803/4292395427]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-09T21:53:10.176 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:53:10.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:09 smithi090 ceph-mon[36996]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:53:10.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:09 smithi090 ceph-mon[36996]: from='osd.0 [v2:172.21.15.6:6802/4292395427,v1:172.21.15.6:6803/4292395427]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-09T21:53:11.305 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:10 smithi006 ceph-mon[29258]: from='osd.0 [v2:172.21.15.6:6802/4292395427,v1:172.21.15.6:6803/4292395427]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-09T21:53:11.305 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:10 smithi006 ceph-mon[29258]: osdmap e7: 1 total, 0 up, 1 in 2024-02-09T21:53:11.305 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:10 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T21:53:11.305 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:10 smithi006 ceph-mon[29258]: from='osd.0 [v2:172.21.15.6:6802/4292395427,v1:172.21.15.6:6803/4292395427]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-02-09T21:53:11.305 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:10 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:11.305 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:10 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:11.306 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:53:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:53:10.987+0000 7f5b143cc700 -1 osd.0 0 waiting for initial osdmap 2024-02-09T21:53:11.306 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:53:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:53:10.991+0000 7f5b0b9c0700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T21:53:11.430 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:10 smithi090 ceph-mon[36996]: from='osd.0 [v2:172.21.15.6:6802/4292395427,v1:172.21.15.6:6803/4292395427]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-09T21:53:11.430 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:10 smithi090 ceph-mon[36996]: osdmap e7: 1 total, 0 up, 1 in 2024-02-09T21:53:11.430 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:10 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T21:53:11.430 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:10 smithi090 ceph-mon[36996]: from='osd.0 [v2:172.21.15.6:6802/4292395427,v1:172.21.15.6:6803/4292395427]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-02-09T21:53:11.430 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:10 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:11.430 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:10 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:12.248 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:11 smithi006 ceph-mon[29258]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:53:12.248 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:11 smithi006 ceph-mon[29258]: from='osd.0 [v2:172.21.15.6:6802/4292395427,v1:172.21.15.6:6803/4292395427]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-02-09T21:53:12.249 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:11 smithi006 ceph-mon[29258]: osdmap e8: 1 total, 0 up, 1 in 2024-02-09T21:53:12.249 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:11 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T21:53:12.249 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:11 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T21:53:12.249 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:11 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:53:12.454 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:11 smithi090 ceph-mon[36996]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:53:12.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:11 smithi090 ceph-mon[36996]: from='osd.0 [v2:172.21.15.6:6802/4292395427,v1:172.21.15.6:6803/4292395427]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-02-09T21:53:12.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:11 smithi090 ceph-mon[36996]: osdmap e8: 1 total, 0 up, 1 in 2024-02-09T21:53:12.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:11 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T21:53:12.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:11 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T21:53:12.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:11 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:53:12.695 INFO:teuthology.orchestra.run.smithi006.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-09T21:53:12.696 INFO:teuthology.orchestra.run.smithi006.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-09T21:53:12.696 INFO:teuthology.orchestra.run.smithi006.stderr: stderr: 10+0 records in 2024-02-09T21:53:12.696 INFO:teuthology.orchestra.run.smithi006.stderr:10+0 records out 2024-02-09T21:53:12.696 INFO:teuthology.orchestra.run.smithi006.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125244 s, 837 MB/s 2024-02-09T21:53:12.696 INFO:teuthology.orchestra.run.smithi006.stderr:--> Zapping successful for: 2024-02-09T21:53:13.314 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:13 smithi006 ceph-mon[29258]: purged_snaps scrub starts 2024-02-09T21:53:13.314 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:13 smithi006 ceph-mon[29258]: purged_snaps scrub ok 2024-02-09T21:53:13.314 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:13 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T21:53:13.314 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:13 smithi006 ceph-mon[29258]: osd.0 [v2:172.21.15.6:6802/4292395427,v1:172.21.15.6:6803/4292395427] boot 2024-02-09T21:53:13.314 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:13 smithi006 ceph-mon[29258]: osdmap e9: 1 total, 1 up, 1 in 2024-02-09T21:53:13.315 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:13 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T21:53:13.454 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:13 smithi090 ceph-mon[36996]: purged_snaps scrub starts 2024-02-09T21:53:13.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:13 smithi090 ceph-mon[36996]: purged_snaps scrub ok 2024-02-09T21:53:13.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:13 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T21:53:13.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:13 smithi090 ceph-mon[36996]: osd.0 [v2:172.21.15.6:6802/4292395427,v1:172.21.15.6:6803/4292395427] boot 2024-02-09T21:53:13.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:13 smithi090 ceph-mon[36996]: osdmap e9: 1 total, 1 up, 1 in 2024-02-09T21:53:13.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:13 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T21:53:13.573 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi006:vg_nvme/lv_3 2024-02-09T21:53:13.916 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:53:14.321 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:14 smithi006 ceph-mon[29258]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:53:14.321 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:14 smithi006 ceph-mon[29258]: osdmap e10: 1 total, 1 up, 1 in 2024-02-09T21:53:14.454 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:14 smithi090 ceph-mon[36996]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T21:53:14.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:14 smithi090 ceph-mon[36996]: osdmap e10: 1 total, 1 up, 1 in 2024-02-09T21:53:15.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.662+0000 7f977d6f0700 1 -- 172.21.15.6:0/2010299338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97781003a0 msgr2=0x7f9778102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:53:15.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.662+0000 7f977d6f0700 1 --2- 172.21.15.6:0/2010299338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97781003a0 0x7f9778102780 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f976c004940 tx=0x7f976c009e30 comp rx=0 tx=0).stop 2024-02-09T21:53:15.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.662+0000 7f977d6f0700 1 -- 172.21.15.6:0/2010299338 shutdown_connections 2024-02-09T21:53:15.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.662+0000 7f977d6f0700 1 --2- 172.21.15.6:0/2010299338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97781003a0 0x7f9778102780 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:53:15.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.662+0000 7f977d6f0700 1 --2- 172.21.15.6:0/2010299338 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97780fda80 0x7f97780ffe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:53:15.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.662+0000 7f977d6f0700 1 -- 172.21.15.6:0/2010299338 >> 172.21.15.6:0/2010299338 conn(0x7f97780f7520 msgr2=0x7f97780f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:53:15.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.662+0000 7f977d6f0700 1 -- 172.21.15.6:0/2010299338 shutdown_connections 2024-02-09T21:53:15.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.662+0000 7f977d6f0700 1 -- 172.21.15.6:0/2010299338 wait complete. 2024-02-09T21:53:15.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.662+0000 7f977d6f0700 1 Processor -- start 2024-02-09T21:53:15.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.662+0000 7f977d6f0700 1 -- start start 2024-02-09T21:53:15.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.662+0000 7f977d6f0700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97780fda80 0x7f977810d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:53:15.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.663+0000 7f977d6f0700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97781003a0 0x7f977810de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:53:15.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.663+0000 7f977d6f0700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977810e490 con 0x7f97780fda80 2024-02-09T21:53:15.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.663+0000 7f977d6f0700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977810ed30 con 0x7f97781003a0 2024-02-09T21:53:15.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.663+0000 7f9776ffd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97780fda80 0x7f977810d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:53:15.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.663+0000 7f9776ffd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97780fda80 0x7f977810d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57736/0 (socket says 172.21.15.6:57736) 2024-02-09T21:53:15.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.663+0000 7f9776ffd700 1 -- 172.21.15.6:0/1045269118 learned_addr learned my addr 172.21.15.6:0/1045269118 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:53:15.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.663+0000 7f97767fc700 1 --2- 172.21.15.6:0/1045269118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97781003a0 0x7f977810de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:53:15.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.663+0000 7f9776ffd700 1 -- 172.21.15.6:0/1045269118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97781003a0 msgr2=0x7f977810de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:53:15.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.663+0000 7f9776ffd700 1 --2- 172.21.15.6:0/1045269118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97781003a0 0x7f977810de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:53:15.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.663+0000 7f9776ffd700 1 -- 172.21.15.6:0/1045269118 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f976c005040 con 0x7f97780fda80 2024-02-09T21:53:15.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.663+0000 7f9776ffd700 1 --2- 172.21.15.6:0/1045269118 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97780fda80 0x7f977810d950 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f976800c830 tx=0x7f9768004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:53:15.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.663+0000 7f975f7fe700 1 -- 172.21.15.6:0/1045269118 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97680106f0 con 0x7f97780fda80 2024-02-09T21:53:15.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.663+0000 7f977d6f0700 1 -- 172.21.15.6:0/1045269118 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f977810f010 con 0x7f97780fda80 2024-02-09T21:53:15.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.663+0000 7f975f7fe700 1 -- 172.21.15.6:0/1045269118 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9768010d20 con 0x7f97780fda80 2024-02-09T21:53:15.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.663+0000 7f975f7fe700 1 -- 172.21.15.6:0/1045269118 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97680113a0 con 0x7f97780fda80 2024-02-09T21:53:15.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.663+0000 7f977d6f0700 1 -- 172.21.15.6:0/1045269118 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f977810f550 con 0x7f97780fda80 2024-02-09T21:53:15.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.664+0000 7f977d6f0700 1 -- 172.21.15.6:0/1045269118 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f977804bed0 con 0x7f97780fda80 2024-02-09T21:53:15.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.665+0000 7f975f7fe700 1 -- 172.21.15.6:0/1045269118 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 18) v1 ==== 86563+0+0 (secure 0 0 0) 0x7f9768012400 con 0x7f97780fda80 2024-02-09T21:53:15.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.665+0000 7f975f7fe700 1 --2- 172.21.15.6:0/1045269118 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f9760069210 0x7f976006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:53:15.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.665+0000 7f975f7fe700 1 -- 172.21.15.6:0/1045269118 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7f9768086280 con 0x7f97780fda80 2024-02-09T21:53:15.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.665+0000 7f97767fc700 1 --2- 172.21.15.6:0/1045269118 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f9760069210 0x7f976006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:53:15.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.665+0000 7f97767fc700 1 --2- 172.21.15.6:0/1045269118 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f9760069210 0x7f976006b6c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f976c00b3b0 tx=0x7f976c009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:53:15.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.669+0000 7f975f7fe700 1 -- 172.21.15.6:0/1045269118 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9768056ec0 con 0x7f97780fda80 2024-02-09T21:53:15.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:15.873+0000 7f977d6f0700 1 -- 172.21.15.6:0/1045269118 --> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f977810fa20 con 0x7f9760069210 2024-02-09T21:53:16.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:16 smithi006 ceph-mon[29258]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-09T21:53:16.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:16 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T21:53:16.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:16 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T21:53:16.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:16 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:53:16.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:16 smithi090 ceph-mon[36996]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-09T21:53:16.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:16 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T21:53:16.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:16 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T21:53:16.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:16 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:53:17.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:17 smithi006 ceph-mon[29258]: from='client.14266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:53:17.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:17 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:17.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:17 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:17.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:17 smithi090 ceph-mon[36996]: from='client.14266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:53:17.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:17 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:17.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:17 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:18.249 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:18 smithi006 ceph-mon[29258]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-09T21:53:18.454 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:18 smithi090 ceph-mon[36996]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-09T21:53:20.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:20 smithi006 ceph-mon[29258]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-09T21:53:20.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:20 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/52250045' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2257408-632e-4e27-ae69-99046b12b405"}]: dispatch 2024-02-09T21:53:20.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:20 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/52250045' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2257408-632e-4e27-ae69-99046b12b405"}]': finished 2024-02-09T21:53:20.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:20 smithi006 ceph-mon[29258]: osdmap e11: 2 total, 1 up, 2 in 2024-02-09T21:53:20.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:20 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T21:53:20.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:20 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/537585568' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T21:53:20.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:20 smithi090 ceph-mon[36996]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-09T21:53:20.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:20 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/52250045' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2257408-632e-4e27-ae69-99046b12b405"}]: dispatch 2024-02-09T21:53:20.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:20 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/52250045' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2257408-632e-4e27-ae69-99046b12b405"}]': finished 2024-02-09T21:53:20.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:20 smithi090 ceph-mon[36996]: osdmap e11: 2 total, 1 up, 2 in 2024-02-09T21:53:20.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:20 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T21:53:20.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:20 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/537585568' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T21:53:22.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:22 smithi006 ceph-mon[29258]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-09T21:53:22.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:22 smithi090 ceph-mon[36996]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-09T21:53:24.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:24 smithi006 ceph-mon[29258]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-09T21:53:24.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:24 smithi090 ceph-mon[36996]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-09T21:53:26.202 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:26 smithi006 ceph-mon[29258]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-09T21:53:26.202 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:26 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:26.202 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:26 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:26.202 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:26 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T21:53:26.202 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:26 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:26.202 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:26 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:53:26.202 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:26 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:53:26.202 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:26 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:26.202 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:26 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:53:26.393 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:26 smithi090 ceph-mon[36996]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-09T21:53:26.393 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:26 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:26.393 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:26 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:26.393 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:26 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T21:53:26.393 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:26 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:26.393 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:26 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:53:26.393 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:26 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:53:26.393 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:26 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:26.393 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:26 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:53:27.299 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:27 smithi006 ceph-mon[29258]: Detected new or changed devices on smithi006 2024-02-09T21:53:27.299 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:27 smithi006 ceph-mon[29258]: Adjusting osd_memory_target on smithi006 to 12869M 2024-02-09T21:53:27.299 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:27 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:53:27.454 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:27 smithi090 ceph-mon[36996]: Detected new or changed devices on smithi006 2024-02-09T21:53:27.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:27 smithi090 ceph-mon[36996]: Adjusting osd_memory_target on smithi006 to 12869M 2024-02-09T21:53:27.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:27 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:53:28.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:28 smithi006 ceph-mon[29258]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-09T21:53:28.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:28 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-09T21:53:28.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:28 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:53:28.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:28 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:28.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:28 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:28.454 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:28 smithi090 ceph-mon[36996]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-09T21:53:28.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:28 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-09T21:53:28.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:28 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:53:28.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:28 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:28.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:28 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:29.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:29 smithi006 ceph-mon[29258]: Deploying daemon osd.1 on smithi006 2024-02-09T21:53:29.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:29 smithi090 ceph-mon[36996]: Deploying daemon osd.1 on smithi006 2024-02-09T21:53:30.276 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:30 smithi006 ceph-mon[29258]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-09T21:53:30.356 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:30 smithi090 ceph-mon[36996]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-09T21:53:32.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:32 smithi006 ceph-mon[29258]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-09T21:53:32.454 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:32 smithi090 ceph-mon[36996]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-09T21:53:34.137 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:33 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:34.137 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:33 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:34.137 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:33 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:53:34.137 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:33 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:53:34.137 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:33 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:34.137 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:33 smithi006 ceph-mon[29258]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-09T21:53:34.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:33 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:34.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:33 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:34.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:33 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:53:34.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:33 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:53:34.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:33 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:34.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:33 smithi090 ceph-mon[36996]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-09T21:53:35.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:35 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:35.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:35 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:35.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:35 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:53:35.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:35 smithi006 ceph-mon[29258]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-09T21:53:35.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:35 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:35.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:35 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:35.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:35 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:53:35.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:35 smithi090 ceph-mon[36996]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-09T21:53:37.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:37 smithi006 ceph-mon[29258]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-09T21:53:37.704 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:37 smithi090 ceph-mon[36996]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-09T21:53:37.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:37.743+0000 7f975f7fe700 1 -- 172.21.15.6:0/1045269118 <== mgr.14216 v2:172.21.15.6:6800/2624137529 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f977810fa20 con 0x7f9760069210 2024-02-09T21:53:37.748 INFO:teuthology.orchestra.run.smithi006.stdout:Created osd(s) 1 on host 'smithi006' 2024-02-09T21:53:37.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:37.746+0000 7f977d6f0700 1 -- 172.21.15.6:0/1045269118 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f9760069210 msgr2=0x7f976006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:53:37.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:37.746+0000 7f977d6f0700 1 --2- 172.21.15.6:0/1045269118 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f9760069210 0x7f976006b6c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f976c00b3b0 tx=0x7f976c009e30 comp rx=0 tx=0).stop 2024-02-09T21:53:37.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:37.746+0000 7f977d6f0700 1 -- 172.21.15.6:0/1045269118 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97780fda80 msgr2=0x7f977810d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:53:37.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:37.746+0000 7f977d6f0700 1 --2- 172.21.15.6:0/1045269118 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97780fda80 0x7f977810d950 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f976800c830 tx=0x7f9768004470 comp rx=0 tx=0).stop 2024-02-09T21:53:37.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:37.746+0000 7f977d6f0700 1 -- 172.21.15.6:0/1045269118 shutdown_connections 2024-02-09T21:53:37.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:37.746+0000 7f977d6f0700 1 --2- 172.21.15.6:0/1045269118 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f9760069210 0x7f976006b6c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:53:37.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:37.746+0000 7f977d6f0700 1 --2- 172.21.15.6:0/1045269118 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97780fda80 0x7f977810d950 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:53:37.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:37.746+0000 7f977d6f0700 1 --2- 172.21.15.6:0/1045269118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97781003a0 0x7f977810de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:53:37.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:37.746+0000 7f977d6f0700 1 -- 172.21.15.6:0/1045269118 >> 172.21.15.6:0/1045269118 conn(0x7f97780f7520 msgr2=0x7f97780f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:53:37.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:37.747+0000 7f977d6f0700 1 -- 172.21.15.6:0/1045269118 shutdown_connections 2024-02-09T21:53:37.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:37.747+0000 7f977d6f0700 1 -- 172.21.15.6:0/1045269118 wait complete. 2024-02-09T21:53:38.364 DEBUG:teuthology.orchestra.run.smithi006:osd.1> sudo journalctl -f -n 0 -u ceph-ee26d820-c794-11ee-95b8-87774f69a715@osd.1.service 2024-02-09T21:53:38.367 INFO:tasks.cephadm:Deploying osd.2 on smithi006 with /dev/vg_nvme/lv_2... 2024-02-09T21:53:38.367 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-09T21:53:38.608 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:53:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:53:38.477+0000 7f8351aee540 -1 osd.1 0 log_to_monitors true 2024-02-09T21:53:38.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:38 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:38.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:38 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:38.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:38 smithi006 ceph-mon[29258]: from='osd.1 [v2:172.21.15.6:6810/3556298518,v1:172.21.15.6:6811/3556298518]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-09T21:53:39.045 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:53:39.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:38 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:39.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:38 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:39.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:38 smithi090 ceph-mon[36996]: from='osd.1 [v2:172.21.15.6:6810/3556298518,v1:172.21.15.6:6811/3556298518]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-09T21:53:40.063 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:53:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:53:39.743+0000 7f83482c0700 -1 osd.1 0 waiting for initial osdmap 2024-02-09T21:53:40.064 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:53:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:53:39.747+0000 7f83430bb700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T21:53:40.064 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:39 smithi006 ceph-mon[29258]: from='osd.1 [v2:172.21.15.6:6810/3556298518,v1:172.21.15.6:6811/3556298518]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-09T21:53:40.064 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:39 smithi006 ceph-mon[29258]: osdmap e12: 2 total, 1 up, 2 in 2024-02-09T21:53:40.064 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:39 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T21:53:40.064 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:39 smithi006 ceph-mon[29258]: from='osd.1 [v2:172.21.15.6:6810/3556298518,v1:172.21.15.6:6811/3556298518]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-02-09T21:53:40.064 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:39 smithi006 ceph-mon[29258]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-09T21:53:40.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:39 smithi090 ceph-mon[36996]: from='osd.1 [v2:172.21.15.6:6810/3556298518,v1:172.21.15.6:6811/3556298518]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-09T21:53:40.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:39 smithi090 ceph-mon[36996]: osdmap e12: 2 total, 1 up, 2 in 2024-02-09T21:53:40.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:39 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T21:53:40.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:39 smithi090 ceph-mon[36996]: from='osd.1 [v2:172.21.15.6:6810/3556298518,v1:172.21.15.6:6811/3556298518]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-02-09T21:53:40.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:39 smithi090 ceph-mon[36996]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-09T21:53:41.003 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:40 smithi006 ceph-mon[29258]: from='osd.1 [v2:172.21.15.6:6810/3556298518,v1:172.21.15.6:6811/3556298518]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-02-09T21:53:41.003 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:40 smithi006 ceph-mon[29258]: osdmap e13: 2 total, 1 up, 2 in 2024-02-09T21:53:41.003 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:40 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T21:53:41.003 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:40 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T21:53:41.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:40 smithi090 ceph-mon[36996]: from='osd.1 [v2:172.21.15.6:6810/3556298518,v1:172.21.15.6:6811/3556298518]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-02-09T21:53:41.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:40 smithi090 ceph-mon[36996]: osdmap e13: 2 total, 1 up, 2 in 2024-02-09T21:53:41.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:40 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T21:53:41.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:40 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T21:53:41.279 INFO:teuthology.orchestra.run.smithi006.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-09T21:53:41.279 INFO:teuthology.orchestra.run.smithi006.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-09T21:53:41.279 INFO:teuthology.orchestra.run.smithi006.stderr: stderr: 10+0 records in 2024-02-09T21:53:41.280 INFO:teuthology.orchestra.run.smithi006.stderr:10+0 records out 2024-02-09T21:53:41.280 INFO:teuthology.orchestra.run.smithi006.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0138303 s, 758 MB/s 2024-02-09T21:53:41.280 INFO:teuthology.orchestra.run.smithi006.stderr:--> Zapping successful for: 2024-02-09T21:53:41.861 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:41 smithi006 ceph-mon[29258]: purged_snaps scrub starts 2024-02-09T21:53:41.861 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:41 smithi006 ceph-mon[29258]: purged_snaps scrub ok 2024-02-09T21:53:41.861 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:41 smithi006 ceph-mon[29258]: osd.1 [v2:172.21.15.6:6810/3556298518,v1:172.21.15.6:6811/3556298518] boot 2024-02-09T21:53:41.861 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:41 smithi006 ceph-mon[29258]: osdmap e14: 2 total, 2 up, 2 in 2024-02-09T21:53:41.861 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:41 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T21:53:41.861 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:41 smithi006 ceph-mon[29258]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-09T21:53:41.862 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi006:vg_nvme/lv_2 2024-02-09T21:53:42.041 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:41 smithi090 ceph-mon[36996]: purged_snaps scrub starts 2024-02-09T21:53:42.041 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:41 smithi090 ceph-mon[36996]: purged_snaps scrub ok 2024-02-09T21:53:42.042 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:41 smithi090 ceph-mon[36996]: osd.1 [v2:172.21.15.6:6810/3556298518,v1:172.21.15.6:6811/3556298518] boot 2024-02-09T21:53:42.042 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:41 smithi090 ceph-mon[36996]: osdmap e14: 2 total, 2 up, 2 in 2024-02-09T21:53:42.042 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:41 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T21:53:42.042 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:41 smithi090 ceph-mon[36996]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-09T21:53:42.212 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:53:42.950 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:42 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:53:42.951 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:42 smithi006 ceph-mon[29258]: osdmap e15: 2 total, 2 up, 2 in 2024-02-09T21:53:42.954 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:42 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:53:42.954 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:42 smithi090 ceph-mon[36996]: osdmap e15: 2 total, 2 up, 2 in 2024-02-09T21:53:44.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:43 smithi006 ceph-mon[29258]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-09T21:53:44.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:43 smithi090 ceph-mon[36996]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-09T21:53:44.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.767+0000 7f7c61af0700 1 -- 172.21.15.6:0/1989747 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c5c1000e0 msgr2=0x7f7c5c102520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:53:44.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.767+0000 7f7c61af0700 1 --2- 172.21.15.6:0/1989747 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c5c1000e0 0x7f7c5c102520 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7c44004940 tx=0x7f7c44009e30 comp rx=0 tx=0).stop 2024-02-09T21:53:44.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.767+0000 7f7c61af0700 1 -- 172.21.15.6:0/1989747 shutdown_connections 2024-02-09T21:53:44.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.767+0000 7f7c61af0700 1 --2- 172.21.15.6:0/1989747 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c5c1000e0 0x7f7c5c102520 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:53:44.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.767+0000 7f7c61af0700 1 --2- 172.21.15.6:0/1989747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c5c066c50 0x7f7c5c0ffb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:53:44.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.767+0000 7f7c61af0700 1 -- 172.21.15.6:0/1989747 >> 172.21.15.6:0/1989747 conn(0x7f7c5c0f7520 msgr2=0x7f7c5c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:53:44.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.767+0000 7f7c61af0700 1 -- 172.21.15.6:0/1989747 shutdown_connections 2024-02-09T21:53:44.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.767+0000 7f7c61af0700 1 -- 172.21.15.6:0/1989747 wait complete. 2024-02-09T21:53:44.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.767+0000 7f7c61af0700 1 Processor -- start 2024-02-09T21:53:44.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.767+0000 7f7c61af0700 1 -- start start 2024-02-09T21:53:44.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.767+0000 7f7c61af0700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c5c066c50 0x7f7c5c0fd590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:53:44.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.768+0000 7f7c61af0700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c5c1000e0 0x7f7c5c0fba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:53:44.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.768+0000 7f7c61af0700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c0fc010 con 0x7f7c5c1000e0 2024-02-09T21:53:44.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.768+0000 7f7c61af0700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c0fc180 con 0x7f7c5c066c50 2024-02-09T21:53:44.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.768+0000 7f7c5affd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c5c1000e0 0x7f7c5c0fba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:53:44.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.768+0000 7f7c5affd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c5c1000e0 0x7f7c5c0fba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38090/0 (socket says 172.21.15.6:38090) 2024-02-09T21:53:44.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.768+0000 7f7c5affd700 1 -- 172.21.15.6:0/1470310718 learned_addr learned my addr 172.21.15.6:0/1470310718 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:53:44.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.768+0000 7f7c5b7fe700 1 --2- 172.21.15.6:0/1470310718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c5c066c50 0x7f7c5c0fd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:53:44.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.768+0000 7f7c5affd700 1 -- 172.21.15.6:0/1470310718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c5c066c50 msgr2=0x7f7c5c0fd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:53:44.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.768+0000 7f7c5affd700 1 --2- 172.21.15.6:0/1470310718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c5c066c50 0x7f7c5c0fd590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:53:44.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.768+0000 7f7c5affd700 1 -- 172.21.15.6:0/1470310718 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c44005040 con 0x7f7c5c1000e0 2024-02-09T21:53:44.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.768+0000 7f7c5affd700 1 --2- 172.21.15.6:0/1470310718 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c5c1000e0 0x7f7c5c0fba10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7c4400b3b0 tx=0x7f7c44009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:53:44.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.768+0000 7f7c53fff700 1 -- 172.21.15.6:0/1470310718 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c4400e040 con 0x7f7c5c1000e0 2024-02-09T21:53:44.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.768+0000 7f7c61af0700 1 -- 172.21.15.6:0/1470310718 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c5c0fc400 con 0x7f7c5c1000e0 2024-02-09T21:53:44.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.768+0000 7f7c53fff700 1 -- 172.21.15.6:0/1470310718 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7c4400bcc0 con 0x7f7c5c1000e0 2024-02-09T21:53:44.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.768+0000 7f7c53fff700 1 -- 172.21.15.6:0/1470310718 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c4401c630 con 0x7f7c5c1000e0 2024-02-09T21:53:44.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.768+0000 7f7c61af0700 1 -- 172.21.15.6:0/1470310718 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c5c115980 con 0x7f7c5c1000e0 2024-02-09T21:53:44.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.769+0000 7f7c61af0700 1 -- 172.21.15.6:0/1470310718 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c5c04bed0 con 0x7f7c5c1000e0 2024-02-09T21:53:44.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.770+0000 7f7c53fff700 1 -- 172.21.15.6:0/1470310718 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 18) v1 ==== 86563+0+0 (secure 0 0 0) 0x7f7c44013980 con 0x7f7c5c1000e0 2024-02-09T21:53:44.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.770+0000 7f7c53fff700 1 --2- 172.21.15.6:0/1470310718 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f7c48069210 0x7f7c4806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:53:44.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.770+0000 7f7c53fff700 1 -- 172.21.15.6:0/1470310718 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7f7c44087940 con 0x7f7c5c1000e0 2024-02-09T21:53:44.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.770+0000 7f7c5b7fe700 1 --2- 172.21.15.6:0/1470310718 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f7c48069210 0x7f7c4806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:53:44.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.771+0000 7f7c5b7fe700 1 --2- 172.21.15.6:0/1470310718 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f7c48069210 0x7f7c4806b6c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c0fd320 tx=0x7f7c4c009040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:53:44.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.775+0000 7f7c53fff700 1 -- 172.21.15.6:0/1470310718 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7c44058480 con 0x7f7c5c1000e0 2024-02-09T21:53:44.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:53:44.981+0000 7f7c61af0700 1 -- 172.21.15.6:0/1470310718 --> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f7c5c100d60 con 0x7f7c48069210 2024-02-09T21:53:45.138 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:44 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:45.138 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:44 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:45.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:44 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:45.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:44 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:46.203 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:45 smithi006 ceph-mon[29258]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-09T21:53:46.203 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:45 smithi006 ceph-mon[29258]: from='client.14284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:53:46.203 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:45 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T21:53:46.203 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:45 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T21:53:46.203 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:45 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:53:46.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:45 smithi090 ceph-mon[36996]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-09T21:53:46.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:45 smithi090 ceph-mon[36996]: from='client.14284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:53:46.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:45 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T21:53:46.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:45 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T21:53:46.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:45 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:53:47.653 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:47 smithi006 ceph-mon[29258]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-09T21:53:47.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:47 smithi090 ceph-mon[36996]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-09T21:53:48.856 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:48 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:48.856 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:48 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:48.856 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:48 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T21:53:48.856 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:48 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-09T21:53:48.856 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:48 smithi006 ceph-mon[29258]: Adjusting osd_memory_target on smithi006 to 6434M 2024-02-09T21:53:48.856 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:48 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:48.856 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:48 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:53:48.856 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:48 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:53:48.856 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:48 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:48.857 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:48 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:53:49.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:48 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:49.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:48 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:49.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:48 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T21:53:49.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:48 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-09T21:53:49.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:48 smithi090 ceph-mon[36996]: Adjusting osd_memory_target on smithi006 to 6434M 2024-02-09T21:53:49.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:48 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:49.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:48 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:53:49.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:48 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:53:49.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:48 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:49.206 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:48 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:53:50.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:49 smithi006 ceph-mon[29258]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-09T21:53:50.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:49 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/2402706197' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e56e4fc0-614d-48e6-a2d6-758c34487540"}]: dispatch 2024-02-09T21:53:50.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:49 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/2402706197' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e56e4fc0-614d-48e6-a2d6-758c34487540"}]': finished 2024-02-09T21:53:50.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:49 smithi006 ceph-mon[29258]: osdmap e16: 3 total, 2 up, 3 in 2024-02-09T21:53:50.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:49 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T21:53:50.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:49 smithi090 ceph-mon[36996]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-09T21:53:50.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:49 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/2402706197' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e56e4fc0-614d-48e6-a2d6-758c34487540"}]: dispatch 2024-02-09T21:53:50.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:49 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/2402706197' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e56e4fc0-614d-48e6-a2d6-758c34487540"}]': finished 2024-02-09T21:53:50.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:49 smithi090 ceph-mon[36996]: osdmap e16: 3 total, 2 up, 3 in 2024-02-09T21:53:50.206 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:49 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T21:53:51.048 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:50 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/2713475501' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T21:53:51.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:50 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/2713475501' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T21:53:51.993 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:51 smithi006 ceph-mon[29258]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-09T21:53:52.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:51 smithi090 ceph-mon[36996]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-09T21:53:53.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:53 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:53.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:53 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:53.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:53 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:53:53.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:53 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:53:53.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:53 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:53.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:53 smithi006 ceph-mon[29258]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-09T21:53:53.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:53 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:53.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:53 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:53.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:53 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:53:53.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:53 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:53:53.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:53 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:53:53.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:53 smithi090 ceph-mon[36996]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-09T21:53:55.940 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:55 smithi006 ceph-mon[29258]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-09T21:53:56.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:55 smithi090 ceph-mon[36996]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-09T21:53:57.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:57 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-09T21:53:57.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:57 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:53:57.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:57 smithi006 ceph-mon[29258]: Deploying daemon osd.2 on smithi006 2024-02-09T21:53:57.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:57 smithi006 ceph-mon[29258]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-09T21:53:57.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:57 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:53:57.704 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:57 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-09T21:53:57.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:57 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:53:57.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:57 smithi090 ceph-mon[36996]: Deploying daemon osd.2 on smithi006 2024-02-09T21:53:57.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:57 smithi090 ceph-mon[36996]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-09T21:53:57.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:57 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:54:00.025 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:53:59 smithi006 ceph-mon[29258]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-09T21:54:00.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:53:59 smithi090 ceph-mon[36996]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-09T21:54:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:01 smithi006 ceph-mon[29258]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-09T21:54:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:54:02.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:01 smithi090 ceph-mon[36996]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-09T21:54:02.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:02.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:02.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:54:04.150 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:03 smithi006 ceph-mon[29258]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-09T21:54:04.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:03 smithi090 ceph-mon[36996]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-09T21:54:05.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:54:05.390+0000 7f7c53fff700 1 -- 172.21.15.6:0/1470310718 <== mgr.14216 v2:172.21.15.6:6800/2624137529 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7c5c100d60 con 0x7f7c48069210 2024-02-09T21:54:05.396 INFO:teuthology.orchestra.run.smithi006.stdout:Created osd(s) 2 on host 'smithi006' 2024-02-09T21:54:05.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:54:05.392+0000 7f7c61af0700 1 -- 172.21.15.6:0/1470310718 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f7c48069210 msgr2=0x7f7c4806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:54:05.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:54:05.392+0000 7f7c61af0700 1 --2- 172.21.15.6:0/1470310718 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f7c48069210 0x7f7c4806b6c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c0fd320 tx=0x7f7c4c009040 comp rx=0 tx=0).stop 2024-02-09T21:54:05.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:54:05.392+0000 7f7c61af0700 1 -- 172.21.15.6:0/1470310718 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c5c1000e0 msgr2=0x7f7c5c0fba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:54:05.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:54:05.392+0000 7f7c61af0700 1 --2- 172.21.15.6:0/1470310718 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c5c1000e0 0x7f7c5c0fba10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7c4400b3b0 tx=0x7f7c44009e30 comp rx=0 tx=0).stop 2024-02-09T21:54:05.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:54:05.392+0000 7f7c61af0700 1 -- 172.21.15.6:0/1470310718 shutdown_connections 2024-02-09T21:54:05.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:54:05.392+0000 7f7c61af0700 1 --2- 172.21.15.6:0/1470310718 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f7c48069210 0x7f7c4806b6c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:54:05.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:54:05.392+0000 7f7c61af0700 1 --2- 172.21.15.6:0/1470310718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c5c066c50 0x7f7c5c0fd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:54:05.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:54:05.392+0000 7f7c61af0700 1 --2- 172.21.15.6:0/1470310718 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c5c1000e0 0x7f7c5c0fba10 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:54:05.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:54:05.392+0000 7f7c61af0700 1 -- 172.21.15.6:0/1470310718 >> 172.21.15.6:0/1470310718 conn(0x7f7c5c0f7520 msgr2=0x7f7c5c0f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:54:05.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:54:05.392+0000 7f7c61af0700 1 -- 172.21.15.6:0/1470310718 shutdown_connections 2024-02-09T21:54:05.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:54:05.392+0000 7f7c61af0700 1 -- 172.21.15.6:0/1470310718 wait complete. 2024-02-09T21:54:06.021 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:05 smithi006 ceph-mon[29258]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-09T21:54:06.022 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:05 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:06.022 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:05 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:06.022 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:05 smithi006 ceph-mon[29258]: from='osd.2 [v2:172.21.15.6:6818/3201508057,v1:172.21.15.6:6819/3201508057]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-09T21:54:06.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:05 smithi090 ceph-mon[36996]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-09T21:54:06.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:05 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:06.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:05 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:06.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:05 smithi090 ceph-mon[36996]: from='osd.2 [v2:172.21.15.6:6818/3201508057,v1:172.21.15.6:6819/3201508057]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-09T21:54:06.280 DEBUG:teuthology.orchestra.run.smithi006:osd.2> sudo journalctl -f -n 0 -u ceph-ee26d820-c794-11ee-95b8-87774f69a715@osd.2.service 2024-02-09T21:54:06.283 INFO:tasks.cephadm:Deploying osd.3 on smithi090 with /dev/vg_nvme/lv_4... 2024-02-09T21:54:06.283 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-09T21:54:06.542 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi090/config 2024-02-09T21:54:07.395 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:07 smithi006 ceph-mon[29258]: from='osd.2 [v2:172.21.15.6:6818/3201508057,v1:172.21.15.6:6819/3201508057]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-09T21:54:07.395 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:07 smithi006 ceph-mon[29258]: osdmap e17: 3 total, 2 up, 3 in 2024-02-09T21:54:07.395 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:07 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T21:54:07.395 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:07 smithi006 ceph-mon[29258]: from='osd.2 [v2:172.21.15.6:6818/3201508057,v1:172.21.15.6:6819/3201508057]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-02-09T21:54:07.395 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:07 smithi006 ceph-mon[29258]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-09T21:54:07.395 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:54:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:54:07.385+0000 7f93c6a08700 -1 osd.2 0 waiting for initial osdmap 2024-02-09T21:54:07.395 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:54:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:54:07.390+0000 7f93c3006700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T21:54:07.664 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:07 smithi090 ceph-mon[36996]: from='osd.2 [v2:172.21.15.6:6818/3201508057,v1:172.21.15.6:6819/3201508057]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-09T21:54:07.664 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:07 smithi090 ceph-mon[36996]: osdmap e17: 3 total, 2 up, 3 in 2024-02-09T21:54:07.664 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:07 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T21:54:07.664 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:07 smithi090 ceph-mon[36996]: from='osd.2 [v2:172.21.15.6:6818/3201508057,v1:172.21.15.6:6819/3201508057]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-02-09T21:54:07.664 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:07 smithi090 ceph-mon[36996]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-09T21:54:08.414 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-09T21:54:08.415 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-09T21:54:08.415 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-02-09T21:54:08.415 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-02-09T21:54:08.415 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115361 s, 909 MB/s 2024-02-09T21:54:08.415 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-02-09T21:54:08.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:08 smithi006 ceph-mon[29258]: from='osd.2 [v2:172.21.15.6:6818/3201508057,v1:172.21.15.6:6819/3201508057]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-02-09T21:54:08.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:08 smithi006 ceph-mon[29258]: osdmap e18: 3 total, 2 up, 3 in 2024-02-09T21:54:08.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:08 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T21:54:08.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:08 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T21:54:08.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:08 smithi090 ceph-mon[36996]: from='osd.2 [v2:172.21.15.6:6818/3201508057,v1:172.21.15.6:6819/3201508057]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-02-09T21:54:08.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:08 smithi090 ceph-mon[36996]: osdmap e18: 3 total, 2 up, 3 in 2024-02-09T21:54:08.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:08 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T21:54:08.706 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:08 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T21:54:08.983 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi090:vg_nvme/lv_4 2024-02-09T21:54:09.230 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi090/config 2024-02-09T21:54:09.657 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:09 smithi090 ceph-mon[36996]: purged_snaps scrub starts 2024-02-09T21:54:09.657 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:09 smithi090 ceph-mon[36996]: purged_snaps scrub ok 2024-02-09T21:54:09.657 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:09 smithi090 ceph-mon[36996]: osd.2 [v2:172.21.15.6:6818/3201508057,v1:172.21.15.6:6819/3201508057] boot 2024-02-09T21:54:09.657 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:09 smithi090 ceph-mon[36996]: osdmap e19: 3 total, 3 up, 3 in 2024-02-09T21:54:09.657 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:09 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T21:54:09.658 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:09 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:09.658 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:09 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:09.658 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:09 smithi090 ceph-mon[36996]: pgmap v73: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-09T21:54:09.658 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:09 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-09T21:54:09.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:09 smithi006 ceph-mon[29258]: purged_snaps scrub starts 2024-02-09T21:54:09.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:09 smithi006 ceph-mon[29258]: purged_snaps scrub ok 2024-02-09T21:54:09.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:09 smithi006 ceph-mon[29258]: osd.2 [v2:172.21.15.6:6818/3201508057,v1:172.21.15.6:6819/3201508057] boot 2024-02-09T21:54:09.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:09 smithi006 ceph-mon[29258]: osdmap e19: 3 total, 3 up, 3 in 2024-02-09T21:54:09.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:09 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T21:54:09.659 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:09 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:09.659 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:09 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:09.659 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:09 smithi006 ceph-mon[29258]: pgmap v73: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-09T21:54:09.659 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:09 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-09T21:54:10.514 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:10 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-09T21:54:10.515 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:10 smithi090 ceph-mon[36996]: osdmap e20: 3 total, 3 up, 3 in 2024-02-09T21:54:10.515 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:10 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T21:54:10.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.712+0000 7f42cc267700 1 -- 172.21.15.90:0/156172874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42c40fcef0 msgr2=0x7f42c40fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:54:10.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.712+0000 7f42cc267700 1 --2- 172.21.15.90:0/156172874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42c40fcef0 0x7f42c40fd320 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f42b4004970 tx=0x7f42b4009e30 comp rx=0 tx=0).stop 2024-02-09T21:54:10.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.712+0000 7f42cc267700 1 -- 172.21.15.90:0/156172874 shutdown_connections 2024-02-09T21:54:10.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.712+0000 7f42cc267700 1 --2- 172.21.15.90:0/156172874 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42c40fe0e0 0x7f42c40fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:54:10.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.712+0000 7f42cc267700 1 --2- 172.21.15.90:0/156172874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42c40fcef0 0x7f42c40fd320 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:54:10.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.712+0000 7f42cc267700 1 -- 172.21.15.90:0/156172874 >> 172.21.15.90:0/156172874 conn(0x7f42c40f8620 msgr2=0x7f42c40faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:54:10.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.712+0000 7f42cc267700 1 -- 172.21.15.90:0/156172874 shutdown_connections 2024-02-09T21:54:10.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.712+0000 7f42cc267700 1 -- 172.21.15.90:0/156172874 wait complete. 2024-02-09T21:54:10.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.712+0000 7f42cc267700 1 Processor -- start 2024-02-09T21:54:10.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.712+0000 7f42cc267700 1 -- start start 2024-02-09T21:54:10.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.713+0000 7f42cc267700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42c40fcef0 0x7f42c4071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:54:10.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.713+0000 7f42cc267700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42c40fe0e0 0x7f42c40702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:54:10.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.713+0000 7f42cc267700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c40708e0 con 0x7f42c40fe0e0 2024-02-09T21:54:10.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.713+0000 7f42cc267700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c4070a50 con 0x7f42c40fcef0 2024-02-09T21:54:10.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.713+0000 7f42ca003700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42c40fcef0 0x7f42c4071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:54:10.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.713+0000 7f42ca003700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42c40fcef0 0x7f42c4071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55404/0 (socket says 172.21.15.90:55404) 2024-02-09T21:54:10.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.713+0000 7f42ca003700 1 -- 172.21.15.90:0/570551374 learned_addr learned my addr 172.21.15.90:0/570551374 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-09T21:54:10.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.713+0000 7f42c9802700 1 --2- 172.21.15.90:0/570551374 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42c40fe0e0 0x7f42c40702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:54:10.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.713+0000 7f42ca003700 1 -- 172.21.15.90:0/570551374 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42c40fe0e0 msgr2=0x7f42c40702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:54:10.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.713+0000 7f42ca003700 1 --2- 172.21.15.90:0/570551374 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42c40fe0e0 0x7f42c40702e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:54:10.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.713+0000 7f42ca003700 1 -- 172.21.15.90:0/570551374 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b4005040 con 0x7f42c40fcef0 2024-02-09T21:54:10.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.713+0000 7f42ca003700 1 --2- 172.21.15.90:0/570551374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42c40fcef0 0x7f42c4071e60 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f42b4005010 tx=0x7f42b4009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:54:10.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.713+0000 7f42baffd700 1 -- 172.21.15.90:0/570551374 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42b400e040 con 0x7f42c40fcef0 2024-02-09T21:54:10.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.713+0000 7f42baffd700 1 -- 172.21.15.90:0/570551374 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f42b4004290 con 0x7f42c40fcef0 2024-02-09T21:54:10.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.713+0000 7f42cc267700 1 -- 172.21.15.90:0/570551374 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42c4070cd0 con 0x7f42c40fcef0 2024-02-09T21:54:10.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.713+0000 7f42baffd700 1 -- 172.21.15.90:0/570551374 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42b401c3f0 con 0x7f42c40fcef0 2024-02-09T21:54:10.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.713+0000 7f42cc267700 1 -- 172.21.15.90:0/570551374 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42c4115920 con 0x7f42c40fcef0 2024-02-09T21:54:10.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.714+0000 7f42b8ff9700 1 -- 172.21.15.90:0/570551374 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42ac0052b0 con 0x7f42c40fcef0 2024-02-09T21:54:10.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.715+0000 7f42baffd700 1 -- 172.21.15.90:0/570551374 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) v1 ==== 86563+0+0 (secure 0 0 0) 0x7f42b40138b0 con 0x7f42c40fcef0 2024-02-09T21:54:10.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.715+0000 7f42baffd700 1 --2- 172.21.15.90:0/570551374 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f42b0069210 0x7f42b006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:54:10.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.715+0000 7f42baffd700 1 -- 172.21.15.90:0/570551374 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3228+0+0 (secure 0 0 0) 0x7f42b4087a90 con 0x7f42c40fcef0 2024-02-09T21:54:10.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.715+0000 7f42c9802700 1 --2- 172.21.15.90:0/570551374 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f42b0069210 0x7f42b006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:54:10.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.716+0000 7f42c9802700 1 --2- 172.21.15.90:0/570551374 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f42b0069210 0x7f42b006b6c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f42c0009b80 tx=0x7f42c000b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:54:10.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.720+0000 7f42baffd700 1 -- 172.21.15.90:0/570551374 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f42b40581f0 con 0x7f42c40fcef0 2024-02-09T21:54:10.900 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:10 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-09T21:54:10.901 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:10 smithi006 ceph-mon[29258]: osdmap e20: 3 total, 3 up, 3 in 2024-02-09T21:54:10.901 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:10 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T21:54:10.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:10.909+0000 7f42b8ff9700 1 -- 172.21.15.90:0/570551374 --> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f42ac000bc0 con 0x7f42b0069210 2024-02-09T21:54:11.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:54:11 smithi006 sudo[68040]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-09T21:54:11.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:54:11 smithi006 sudo[68040]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T21:54:11.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:54:11 smithi006 sudo[68040]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T21:54:11.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:54:11 smithi006 sudo[68040]: pam_unix(sudo:session): session closed for user root 2024-02-09T21:54:11.561 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:54:11 smithi006 sudo[68046]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-09T21:54:11.561 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:54:11 smithi006 sudo[68046]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T21:54:11.561 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:54:11 smithi006 sudo[68046]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T21:54:11.561 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:54:11 smithi006 sudo[68046]: pam_unix(sudo:session): session closed for user root 2024-02-09T21:54:11.561 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:11 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-09T21:54:11.562 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:11 smithi006 ceph-mon[29258]: osdmap e21: 3 total, 3 up, 3 in 2024-02-09T21:54:11.562 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:11 smithi006 ceph-mon[29258]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-09T21:54:11.562 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:11 smithi006 ceph-mon[29258]: from='client.24127 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:54:11.562 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:11 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T21:54:11.562 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:11 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T21:54:11.562 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:11 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:54:11.562 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:11 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:11.636 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:11 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-09T21:54:11.636 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:11 smithi090 ceph-mon[36996]: osdmap e21: 3 total, 3 up, 3 in 2024-02-09T21:54:11.636 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:11 smithi090 ceph-mon[36996]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-09T21:54:11.636 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:11 smithi090 ceph-mon[36996]: from='client.24127 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:54:11.636 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:11 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T21:54:11.636 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:11 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T21:54:11.636 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:11 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:54:11.636 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:11 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:11.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:54:11 smithi006 sudo[68049]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-09T21:54:11.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:54:11 smithi006 sudo[68049]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T21:54:11.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:54:11 smithi006 sudo[68049]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T21:54:11.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:54:11 smithi006 sudo[68049]: pam_unix(sudo:session): session closed for user root 2024-02-09T21:54:12.174 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:54:11 smithi006 sudo[68150]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-09T21:54:12.175 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:54:11 smithi006 sudo[68150]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T21:54:12.175 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:54:11 smithi006 sudo[68150]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T21:54:12.175 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:54:11 smithi006 sudo[68150]: pam_unix(sudo:session): session closed for user root 2024-02-09T21:54:12.438 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:54:12 smithi006 sudo[68183]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-09T21:54:12.439 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:54:12 smithi006 sudo[68183]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T21:54:12.439 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:54:12 smithi006 sudo[68183]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T21:54:12.439 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:54:12 smithi006 sudo[68183]: pam_unix(sudo:session): session closed for user root 2024-02-09T21:54:12.694 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:54:12 smithi006 sudo[68186]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-09T21:54:12.694 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:54:12 smithi006 sudo[68186]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T21:54:12.695 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:54:12 smithi006 sudo[68186]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T21:54:12.695 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:54:12 smithi006 sudo[68186]: pam_unix(sudo:session): session closed for user root 2024-02-09T21:54:12.960 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:12 smithi006 sudo[68189]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-09T21:54:12.960 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:12 smithi006 sudo[68189]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T21:54:12.960 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:12 smithi006 sudo[68189]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T21:54:12.961 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:12 smithi006 sudo[68189]: pam_unix(sudo:session): session closed for user root 2024-02-09T21:54:12.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:12.970+0000 7f42baffd700 1 -- 172.21.15.90:0/570551374 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f42b4054b30 con 0x7f42c40fcef0 2024-02-09T21:54:13.218 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:12 smithi090 ceph-mon[36996]: osdmap e22: 3 total, 3 up, 3 in 2024-02-09T21:54:13.218 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:12 smithi090 ceph-mon[36996]: Detected new or changed devices on smithi006 2024-02-09T21:54:13.218 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:12 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:13.218 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:12 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:13.218 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:12 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T21:54:13.218 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:12 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-09T21:54:13.218 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:12 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-09T21:54:13.219 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:12 smithi090 ceph-mon[36996]: Adjusting osd_memory_target on smithi006 to 4289M 2024-02-09T21:54:13.219 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:12 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:13.219 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:12 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:54:13.219 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:12 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:54:13.219 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:12 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:13.219 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:12 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:54:13.219 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:12 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:54:13.219 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:12 smithi090 ceph-mon[36996]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-09T21:54:13.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:12 smithi006 sudo[68192]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-09T21:54:13.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:12 smithi006 sudo[68192]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T21:54:13.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:12 smithi006 sudo[68192]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T21:54:13.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:12 smithi006 sudo[68192]: pam_unix(sudo:session): session closed for user root 2024-02-09T21:54:13.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:12 smithi006 ceph-mon[29258]: osdmap e22: 3 total, 3 up, 3 in 2024-02-09T21:54:13.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:12 smithi006 ceph-mon[29258]: Detected new or changed devices on smithi006 2024-02-09T21:54:13.409 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:12 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:13.409 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:12 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:13.409 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:12 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T21:54:13.409 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:12 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-09T21:54:13.409 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:12 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-09T21:54:13.409 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:12 smithi006 ceph-mon[29258]: Adjusting osd_memory_target on smithi006 to 4289M 2024-02-09T21:54:13.409 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:12 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:13.409 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:12 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:54:13.409 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:12 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:54:13.409 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:12 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:13.410 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:12 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:54:13.410 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:12 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:54:13.410 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:12 smithi006 ceph-mon[29258]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-09T21:54:13.469 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:13 smithi090 sudo[40386]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-09T21:54:13.470 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:13 smithi090 sudo[40386]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T21:54:13.470 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:13 smithi090 sudo[40386]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T21:54:13.470 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:13 smithi090 sudo[40386]: pam_unix(sudo:session): session closed for user root 2024-02-09T21:54:13.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:13 smithi090 sudo[40389]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-09T21:54:13.956 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:13 smithi090 sudo[40389]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T21:54:13.956 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:13 smithi090 sudo[40389]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T21:54:13.956 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:13 smithi090 sudo[40389]: pam_unix(sudo:session): session closed for user root 2024-02-09T21:54:14.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:13 smithi090 ceph-mon[36996]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-02-09T21:54:14.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:13 smithi090 ceph-mon[36996]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-09T21:54:14.274 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:13 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata", "id": "smithi006"}]: dispatch 2024-02-09T21:54:14.274 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:13 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-02-09T21:54:14.274 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:13 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:54:14.274 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:13 smithi090 ceph-mon[36996]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-09T21:54:14.274 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:13 smithi090 ceph-mon[36996]: mgrmap e19: smithi006.glirxt(active, since 2m), standbys: smithi090.csxosj 2024-02-09T21:54:14.274 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:13 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:54:14.274 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:13 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata", "id": "smithi006"}]: dispatch 2024-02-09T21:54:14.274 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:13 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-02-09T21:54:14.274 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:13 smithi090 ceph-mon[36996]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-09T21:54:14.274 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:13 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:14.274 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:13 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:14.274 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:13 smithi090 ceph-mon[36996]: from='client.? 172.21.15.90:0/1568090262' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b2a19ac-de1e-46f4-a3e0-5a8321672e4e"}]: dispatch 2024-02-09T21:54:14.275 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:13 smithi090 ceph-mon[36996]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b2a19ac-de1e-46f4-a3e0-5a8321672e4e"}]: dispatch 2024-02-09T21:54:14.275 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:13 smithi090 ceph-mon[36996]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b2a19ac-de1e-46f4-a3e0-5a8321672e4e"}]': finished 2024-02-09T21:54:14.275 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:13 smithi090 ceph-mon[36996]: osdmap e23: 4 total, 3 up, 4 in 2024-02-09T21:54:14.275 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:13 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T21:54:14.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:13 smithi006 ceph-mon[29258]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-02-09T21:54:14.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:13 smithi006 ceph-mon[29258]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-09T21:54:14.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:13 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata", "id": "smithi006"}]: dispatch 2024-02-09T21:54:14.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:13 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-02-09T21:54:14.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:13 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:54:14.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:13 smithi006 ceph-mon[29258]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-09T21:54:14.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:13 smithi006 ceph-mon[29258]: mgrmap e19: smithi006.glirxt(active, since 2m), standbys: smithi090.csxosj 2024-02-09T21:54:14.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:13 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:54:14.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:13 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata", "id": "smithi006"}]: dispatch 2024-02-09T21:54:14.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:13 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-02-09T21:54:14.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:13 smithi006 ceph-mon[29258]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-09T21:54:14.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:13 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:14.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:13 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:14.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:13 smithi006 ceph-mon[29258]: from='client.? 172.21.15.90:0/1568090262' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b2a19ac-de1e-46f4-a3e0-5a8321672e4e"}]: dispatch 2024-02-09T21:54:14.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:13 smithi006 ceph-mon[29258]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b2a19ac-de1e-46f4-a3e0-5a8321672e4e"}]: dispatch 2024-02-09T21:54:14.409 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:13 smithi006 ceph-mon[29258]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b2a19ac-de1e-46f4-a3e0-5a8321672e4e"}]': finished 2024-02-09T21:54:14.409 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:13 smithi006 ceph-mon[29258]: osdmap e23: 4 total, 3 up, 4 in 2024-02-09T21:54:14.409 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:13 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T21:54:15.373 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:14 smithi006 ceph-mon[29258]: from='client.? 172.21.15.90:0/4050799161' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T21:54:15.454 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:14 smithi090 ceph-mon[36996]: from='client.? 172.21.15.90:0/4050799161' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T21:54:16.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:15 smithi006 ceph-mon[29258]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-09T21:54:16.454 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:15 smithi090 ceph-mon[36996]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-09T21:54:17.654 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:17 smithi006 ceph-mon[29258]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-02-09T21:54:17.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:17 smithi090 ceph-mon[36996]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-02-09T21:54:19.954 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:19 smithi090 ceph-mon[36996]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-02-09T21:54:20.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:19 smithi006 ceph-mon[29258]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-02-09T21:54:20.954 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:20 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-09T21:54:20.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:20 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:54:21.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:20 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-09T21:54:21.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:20 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:54:22.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:21 smithi006 ceph-mon[29258]: Deploying daemon osd.3 on smithi090 2024-02-09T21:54:22.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:21 smithi006 ceph-mon[29258]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-02-09T21:54:22.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:21 smithi090 ceph-mon[36996]: Deploying daemon osd.3 on smithi090 2024-02-09T21:54:22.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:21 smithi090 ceph-mon[36996]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-02-09T21:54:24.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:23 smithi006 ceph-mon[29258]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-09T21:54:24.202 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:23 smithi090 ceph-mon[36996]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-09T21:54:26.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:25 smithi090 ceph-mon[36996]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.6 KiB/s rd, 70 KiB/s wr, 5 op/s 2024-02-09T21:54:26.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:25 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:26.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:25 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:26.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:25 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:54:26.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:25 smithi006 ceph-mon[29258]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.6 KiB/s rd, 70 KiB/s wr, 5 op/s 2024-02-09T21:54:26.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:25 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:26.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:25 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:26.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:25 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:54:27.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:27 smithi006 ceph-mon[29258]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-02-09T21:54:27.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:27 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:54:27.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:27 smithi090 ceph-mon[36996]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-02-09T21:54:27.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:27 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:54:29.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:29.027+0000 7f42baffd700 1 -- 172.21.15.90:0/570551374 <== mgr.14216 v2:172.21.15.6:6800/2624137529 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f42ac000bc0 con 0x7f42b0069210 2024-02-09T21:54:29.032 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 3 on host 'smithi090' 2024-02-09T21:54:29.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:29.029+0000 7f42b8ff9700 1 -- 172.21.15.90:0/570551374 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f42b0069210 msgr2=0x7f42b006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:54:29.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:29.029+0000 7f42b8ff9700 1 --2- 172.21.15.90:0/570551374 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f42b0069210 0x7f42b006b6c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f42c0009b80 tx=0x7f42c000b040 comp rx=0 tx=0).stop 2024-02-09T21:54:29.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:29.029+0000 7f42b8ff9700 1 -- 172.21.15.90:0/570551374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42c40fcef0 msgr2=0x7f42c4071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:54:29.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:29.029+0000 7f42b8ff9700 1 --2- 172.21.15.90:0/570551374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42c40fcef0 0x7f42c4071e60 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f42b4005010 tx=0x7f42b4009e30 comp rx=0 tx=0).stop 2024-02-09T21:54:29.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:29.029+0000 7f42b8ff9700 1 -- 172.21.15.90:0/570551374 shutdown_connections 2024-02-09T21:54:29.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:29.029+0000 7f42b8ff9700 1 --2- 172.21.15.90:0/570551374 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f42b0069210 0x7f42b006b6c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:54:29.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:29.029+0000 7f42b8ff9700 1 --2- 172.21.15.90:0/570551374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42c40fcef0 0x7f42c4071e60 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:54:29.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:29.029+0000 7f42b8ff9700 1 --2- 172.21.15.90:0/570551374 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42c40fe0e0 0x7f42c40702e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:54:29.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:29.029+0000 7f42b8ff9700 1 -- 172.21.15.90:0/570551374 >> 172.21.15.90:0/570551374 conn(0x7f42c40f8620 msgr2=0x7f42c41013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:54:29.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:29.030+0000 7f42b8ff9700 1 -- 172.21.15.90:0/570551374 shutdown_connections 2024-02-09T21:54:29.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:29.030+0000 7f42b8ff9700 1 -- 172.21.15.90:0/570551374 wait complete. 2024-02-09T21:54:29.966 DEBUG:teuthology.orchestra.run.smithi090:osd.3> sudo journalctl -f -n 0 -u ceph-ee26d820-c794-11ee-95b8-87774f69a715@osd.3.service 2024-02-09T21:54:29.969 INFO:tasks.cephadm:Deploying osd.4 on smithi090 with /dev/vg_nvme/lv_3... 2024-02-09T21:54:29.969 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-09T21:54:30.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:29 smithi090 ceph-mon[36996]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-09T21:54:30.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:29 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:30.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:29 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:30.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:29 smithi090 ceph-mon[36996]: from='osd.3 [v2:172.21.15.90:6800/1692709589,v1:172.21.15.90:6801/1692709589]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-09T21:54:30.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:29 smithi090 ceph-mon[36996]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-09T21:54:30.263 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi090/config 2024-02-09T21:54:30.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:29 smithi006 ceph-mon[29258]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-09T21:54:30.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:29 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:30.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:29 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:30.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:29 smithi006 ceph-mon[29258]: from='osd.3 [v2:172.21.15.90:6800/1692709589,v1:172.21.15.90:6801/1692709589]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-09T21:54:30.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:29 smithi006 ceph-mon[29258]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-09T21:54:31.286 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:31 smithi090 ceph-mon[36996]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-09T21:54:31.286 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:31 smithi090 ceph-mon[36996]: osdmap e24: 4 total, 3 up, 4 in 2024-02-09T21:54:31.286 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:31 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T21:54:31.286 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:31 smithi090 ceph-mon[36996]: from='osd.3 [v2:172.21.15.90:6800/1692709589,v1:172.21.15.90:6801/1692709589]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-09T21:54:31.286 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:31 smithi090 ceph-mon[36996]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-09T21:54:31.286 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:31 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:31.286 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:31 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:31.287 INFO:journalctl@ceph.osd.3.smithi090.stdout:Feb 09 21:54:31 smithi090 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-3[42484]: 2024-02-09T21:54:31.007+0000 7f4f8df2a700 -1 osd.3 0 waiting for initial osdmap 2024-02-09T21:54:31.287 INFO:journalctl@ceph.osd.3.smithi090.stdout:Feb 09 21:54:31 smithi090 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-3[42484]: 2024-02-09T21:54:31.013+0000 7f4f88d25700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T21:54:31.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:31 smithi006 ceph-mon[29258]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-09T21:54:31.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:31 smithi006 ceph-mon[29258]: osdmap e24: 4 total, 3 up, 4 in 2024-02-09T21:54:31.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:31 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T21:54:31.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:31 smithi006 ceph-mon[29258]: from='osd.3 [v2:172.21.15.90:6800/1692709589,v1:172.21.15.90:6801/1692709589]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-09T21:54:31.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:31 smithi006 ceph-mon[29258]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-09T21:54:31.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:31 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:31.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:31 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:32.086 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:32 smithi090 ceph-mon[36996]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-09T21:54:32.086 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:32 smithi090 ceph-mon[36996]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-02-09T21:54:32.086 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:32 smithi090 ceph-mon[36996]: osdmap e25: 4 total, 3 up, 4 in 2024-02-09T21:54:32.086 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:32 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T21:54:32.086 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:32 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T21:54:32.086 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:32 smithi090 ceph-mon[36996]: osd.3 [v2:172.21.15.90:6800/1692709589,v1:172.21.15.90:6801/1692709589] boot 2024-02-09T21:54:32.086 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:32 smithi090 ceph-mon[36996]: osdmap e26: 4 total, 4 up, 4 in 2024-02-09T21:54:32.086 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:32 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T21:54:32.140 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-09T21:54:32.141 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-09T21:54:32.141 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-02-09T21:54:32.141 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-02-09T21:54:32.141 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130954 s, 801 MB/s 2024-02-09T21:54:32.141 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-02-09T21:54:32.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:32 smithi006 ceph-mon[29258]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-09T21:54:32.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:32 smithi006 ceph-mon[29258]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-02-09T21:54:32.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:32 smithi006 ceph-mon[29258]: osdmap e25: 4 total, 3 up, 4 in 2024-02-09T21:54:32.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:32 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T21:54:32.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:32 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T21:54:32.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:32 smithi006 ceph-mon[29258]: osd.3 [v2:172.21.15.90:6800/1692709589,v1:172.21.15.90:6801/1692709589] boot 2024-02-09T21:54:32.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:32 smithi006 ceph-mon[29258]: osdmap e26: 4 total, 4 up, 4 in 2024-02-09T21:54:32.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:32 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T21:54:32.941 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi090:vg_nvme/lv_3 2024-02-09T21:54:33.615 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:33 smithi090 ceph-mon[36996]: purged_snaps scrub starts 2024-02-09T21:54:33.615 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:33 smithi090 ceph-mon[36996]: purged_snaps scrub ok 2024-02-09T21:54:33.615 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:33 smithi090 ceph-mon[36996]: osdmap e27: 4 total, 4 up, 4 in 2024-02-09T21:54:33.615 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:33 smithi090 ceph-mon[36996]: pgmap v93: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-09T21:54:33.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:33 smithi006 ceph-mon[29258]: purged_snaps scrub starts 2024-02-09T21:54:33.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:33 smithi006 ceph-mon[29258]: purged_snaps scrub ok 2024-02-09T21:54:33.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:33 smithi006 ceph-mon[29258]: osdmap e27: 4 total, 4 up, 4 in 2024-02-09T21:54:33.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:33 smithi006 ceph-mon[29258]: pgmap v93: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-09T21:54:33.838 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi090/config 2024-02-09T21:54:34.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:34 smithi006 ceph-mon[29258]: osdmap e28: 4 total, 4 up, 4 in 2024-02-09T21:54:34.704 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:34 smithi090 ceph-mon[36996]: osdmap e28: 4 total, 4 up, 4 in 2024-02-09T21:54:35.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:35 smithi006 ceph-mon[29258]: osdmap e29: 4 total, 4 up, 4 in 2024-02-09T21:54:35.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:35 smithi006 ceph-mon[29258]: pgmap v96: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-09T21:54:35.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:35 smithi006 ceph-mon[29258]: Detected new or changed devices on smithi090 2024-02-09T21:54:35.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:35 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:35.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:35 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:35.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:35 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-09T21:54:35.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:35 smithi006 ceph-mon[29258]: Adjusting osd_memory_target on smithi090 to 15936M 2024-02-09T21:54:35.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:35 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:35.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:35 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:54:35.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:35 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:54:35.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:35 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:35.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:35 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:54:35.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:35 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:54:35.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:35 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:54:35.659 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:35 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:35.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:35 smithi090 ceph-mon[36996]: osdmap e29: 4 total, 4 up, 4 in 2024-02-09T21:54:35.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:35 smithi090 ceph-mon[36996]: pgmap v96: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-09T21:54:35.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:35 smithi090 ceph-mon[36996]: Detected new or changed devices on smithi090 2024-02-09T21:54:35.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:35 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:35.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:35 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:35.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:35 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-09T21:54:35.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:35 smithi090 ceph-mon[36996]: Adjusting osd_memory_target on smithi090 to 15936M 2024-02-09T21:54:35.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:35 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:35.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:35 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:54:35.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:35 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:54:35.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:35 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:35.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:35 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:54:35.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:35 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:54:35.706 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:35 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:54:35.706 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:35 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:36.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.013+0000 7f43aae22700 1 -- 172.21.15.90:0/2910703131 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43a40fcef0 msgr2=0x7f43a40fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:54:36.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.013+0000 7f43aae22700 1 --2- 172.21.15.90:0/2910703131 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43a40fcef0 0x7f43a40fd320 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4398004970 tx=0x7f4398009e30 comp rx=0 tx=0).stop 2024-02-09T21:54:36.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.013+0000 7f43aae22700 1 -- 172.21.15.90:0/2910703131 shutdown_connections 2024-02-09T21:54:36.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.013+0000 7f43aae22700 1 --2- 172.21.15.90:0/2910703131 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43a40fe0e0 0x7f43a40fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:54:36.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.013+0000 7f43aae22700 1 --2- 172.21.15.90:0/2910703131 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43a40fcef0 0x7f43a40fd320 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:54:36.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.013+0000 7f43aae22700 1 -- 172.21.15.90:0/2910703131 >> 172.21.15.90:0/2910703131 conn(0x7f43a40f8620 msgr2=0x7f43a40faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:54:36.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.013+0000 7f43aae22700 1 -- 172.21.15.90:0/2910703131 shutdown_connections 2024-02-09T21:54:36.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.013+0000 7f43aae22700 1 -- 172.21.15.90:0/2910703131 wait complete. 2024-02-09T21:54:36.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.014+0000 7f43aae22700 1 Processor -- start 2024-02-09T21:54:36.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.014+0000 7f43aae22700 1 -- start start 2024-02-09T21:54:36.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.014+0000 7f43aae22700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43a40fcef0 0x7f43a410d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:54:36.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.014+0000 7f43aae22700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43a40fe0e0 0x7f43a410de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:54:36.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.014+0000 7f43aae22700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a410e430 con 0x7f43a40fe0e0 2024-02-09T21:54:36.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.014+0000 7f43a8bbe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43a40fcef0 0x7f43a410d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:54:36.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.014+0000 7f43aae22700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a410ecd0 con 0x7f43a40fcef0 2024-02-09T21:54:36.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.014+0000 7f43a8bbe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43a40fcef0 0x7f43a410d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40814/0 (socket says 172.21.15.90:40814) 2024-02-09T21:54:36.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.014+0000 7f43a8bbe700 1 -- 172.21.15.90:0/2464586981 learned_addr learned my addr 172.21.15.90:0/2464586981 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-09T21:54:36.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.015+0000 7f43a3fff700 1 --2- 172.21.15.90:0/2464586981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43a40fe0e0 0x7f43a410de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:54:36.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.015+0000 7f43a8bbe700 1 -- 172.21.15.90:0/2464586981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43a40fe0e0 msgr2=0x7f43a410de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:54:36.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.015+0000 7f43a8bbe700 1 --2- 172.21.15.90:0/2464586981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43a40fe0e0 0x7f43a410de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:54:36.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.015+0000 7f43a8bbe700 1 -- 172.21.15.90:0/2464586981 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4398005040 con 0x7f43a40fcef0 2024-02-09T21:54:36.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.015+0000 7f43a3fff700 1 --2- 172.21.15.90:0/2464586981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43a40fe0e0 0x7f43a410de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T21:54:36.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.015+0000 7f43a8bbe700 1 --2- 172.21.15.90:0/2464586981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43a40fcef0 0x7f43a410d8f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4398004f50 tx=0x7f4398009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:54:36.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.015+0000 7f43a17fa700 1 -- 172.21.15.90:0/2464586981 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f439800e040 con 0x7f43a40fcef0 2024-02-09T21:54:36.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.015+0000 7f43a17fa700 1 -- 172.21.15.90:0/2464586981 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f439800bce0 con 0x7f43a40fcef0 2024-02-09T21:54:36.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.015+0000 7f43aae22700 1 -- 172.21.15.90:0/2464586981 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43a410ef50 con 0x7f43a40fcef0 2024-02-09T21:54:36.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.015+0000 7f43a17fa700 1 -- 172.21.15.90:0/2464586981 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4398013660 con 0x7f43a40fcef0 2024-02-09T21:54:36.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.015+0000 7f43aae22700 1 -- 172.21.15.90:0/2464586981 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43a410f490 con 0x7f43a40fcef0 2024-02-09T21:54:36.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.016+0000 7f43aae22700 1 -- 172.21.15.90:0/2464586981 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43a404bed0 con 0x7f43a40fcef0 2024-02-09T21:54:36.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.017+0000 7f43a17fa700 1 -- 172.21.15.90:0/2464586981 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f439801ca90 con 0x7f43a40fcef0 2024-02-09T21:54:36.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.017+0000 7f43a17fa700 1 --2- 172.21.15.90:0/2464586981 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f439406d6b0 0x7f439406fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:54:36.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.018+0000 7f43a17fa700 1 -- 172.21.15.90:0/2464586981 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3770+0+0 (secure 0 0 0) 0x7f4398087700 con 0x7f43a40fcef0 2024-02-09T21:54:36.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.018+0000 7f43a3fff700 1 --2- 172.21.15.90:0/2464586981 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f439406d6b0 0x7f439406fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:54:36.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.018+0000 7f43a3fff700 1 --2- 172.21.15.90:0/2464586981 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f439406d6b0 0x7f439406fb60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4390009fd0 tx=0x7f4390009ee0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:54:36.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.022+0000 7f43a17fa700 1 -- 172.21.15.90:0/2464586981 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4398057c10 con 0x7f43a40fcef0 2024-02-09T21:54:36.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:36.212+0000 7f43aae22700 1 -- 172.21.15.90:0/2464586981 --> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f43a4102ac0 con 0x7f439406d6b0 2024-02-09T21:54:36.628 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:36 smithi090 ceph-mon[36996]: osdmap e30: 4 total, 4 up, 4 in 2024-02-09T21:54:36.628 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:36 smithi090 ceph-mon[36996]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:54:36.628 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:36 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T21:54:36.628 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:36 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T21:54:36.628 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:36 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:54:36.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:36 smithi006 ceph-mon[29258]: osdmap e30: 4 total, 4 up, 4 in 2024-02-09T21:54:36.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:36 smithi006 ceph-mon[29258]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:54:36.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:36 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T21:54:36.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:36 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T21:54:36.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:36 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:54:37.534 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:37 smithi090 ceph-mon[36996]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-02-09T21:54:37.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:37 smithi006 ceph-mon[29258]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-02-09T21:54:38.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:38 smithi006 ceph-mon[29258]: from='client.? 172.21.15.90:0/3646759527' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d79e2bb6-cebd-4e3c-8cec-5fac286838e4"}]: dispatch 2024-02-09T21:54:38.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:38 smithi006 ceph-mon[29258]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d79e2bb6-cebd-4e3c-8cec-5fac286838e4"}]: dispatch 2024-02-09T21:54:38.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:38 smithi006 ceph-mon[29258]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d79e2bb6-cebd-4e3c-8cec-5fac286838e4"}]': finished 2024-02-09T21:54:38.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:38 smithi006 ceph-mon[29258]: osdmap e31: 5 total, 4 up, 5 in 2024-02-09T21:54:38.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:38 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T21:54:38.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:38 smithi090 ceph-mon[36996]: from='client.? 172.21.15.90:0/3646759527' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d79e2bb6-cebd-4e3c-8cec-5fac286838e4"}]: dispatch 2024-02-09T21:54:38.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:38 smithi090 ceph-mon[36996]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d79e2bb6-cebd-4e3c-8cec-5fac286838e4"}]: dispatch 2024-02-09T21:54:38.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:38 smithi090 ceph-mon[36996]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d79e2bb6-cebd-4e3c-8cec-5fac286838e4"}]': finished 2024-02-09T21:54:38.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:38 smithi090 ceph-mon[36996]: osdmap e31: 5 total, 4 up, 5 in 2024-02-09T21:54:38.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:38 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T21:54:39.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:39 smithi006 ceph-mon[29258]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-02-09T21:54:39.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:39 smithi006 ceph-mon[29258]: from='client.? 172.21.15.90:0/1790347052' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T21:54:39.954 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:39 smithi090 ceph-mon[36996]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-02-09T21:54:39.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:39 smithi090 ceph-mon[36996]: from='client.? 172.21.15.90:0/1790347052' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T21:54:42.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:41 smithi006 ceph-mon[29258]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-02-09T21:54:42.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:41 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:54:42.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:41 smithi090 ceph-mon[36996]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-02-09T21:54:42.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:41 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:54:44.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:43 smithi006 ceph-mon[29258]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-09T21:54:44.170 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:43 smithi090 ceph-mon[36996]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-09T21:54:45.954 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:45 smithi090 ceph-mon[36996]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-02-09T21:54:45.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:45 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-09T21:54:45.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:45 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:54:46.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:45 smithi006 ceph-mon[29258]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-02-09T21:54:46.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:45 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-09T21:54:46.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:45 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:54:47.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:46 smithi090 ceph-mon[36996]: Deploying daemon osd.4 on smithi090 2024-02-09T21:54:47.308 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:46 smithi006 ceph-mon[29258]: Deploying daemon osd.4 on smithi090 2024-02-09T21:54:48.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:47 smithi090 ceph-mon[36996]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-09T21:54:48.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:47 smithi006 ceph-mon[29258]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-09T21:54:50.158 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:49 smithi090 ceph-mon[36996]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-09T21:54:50.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:49 smithi006 ceph-mon[29258]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-09T21:54:52.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:51 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:52.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:51 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:52.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:51 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:54:52.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:51 smithi090 ceph-mon[36996]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-09T21:54:52.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:51 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:52.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:51 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:52.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:51 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:54:52.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:51 smithi006 ceph-mon[29258]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-09T21:54:54.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:53 smithi006 ceph-mon[29258]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-09T21:54:54.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:53 smithi090 ceph-mon[36996]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-09T21:54:54.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:54.476+0000 7f43a17fa700 1 -- 172.21.15.90:0/2464586981 <== mgr.14216 v2:172.21.15.6:6800/2624137529 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f43a4102ac0 con 0x7f439406d6b0 2024-02-09T21:54:54.480 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 4 on host 'smithi090' 2024-02-09T21:54:54.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:54.478+0000 7f43aae22700 1 -- 172.21.15.90:0/2464586981 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f439406d6b0 msgr2=0x7f439406fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:54:54.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:54.478+0000 7f43aae22700 1 --2- 172.21.15.90:0/2464586981 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f439406d6b0 0x7f439406fb60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4390009fd0 tx=0x7f4390009ee0 comp rx=0 tx=0).stop 2024-02-09T21:54:54.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:54.478+0000 7f43aae22700 1 -- 172.21.15.90:0/2464586981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43a40fcef0 msgr2=0x7f43a410d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:54:54.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:54.478+0000 7f43aae22700 1 --2- 172.21.15.90:0/2464586981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43a40fcef0 0x7f43a410d8f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4398004f50 tx=0x7f4398009e30 comp rx=0 tx=0).stop 2024-02-09T21:54:54.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:54.478+0000 7f43aae22700 1 -- 172.21.15.90:0/2464586981 shutdown_connections 2024-02-09T21:54:54.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:54.478+0000 7f43aae22700 1 --2- 172.21.15.90:0/2464586981 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f439406d6b0 0x7f439406fb60 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:54:54.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:54.478+0000 7f43aae22700 1 --2- 172.21.15.90:0/2464586981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43a40fcef0 0x7f43a410d8f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:54:54.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:54.478+0000 7f43aae22700 1 --2- 172.21.15.90:0/2464586981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43a40fe0e0 0x7f43a410de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:54:54.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:54.478+0000 7f43aae22700 1 -- 172.21.15.90:0/2464586981 >> 172.21.15.90:0/2464586981 conn(0x7f43a40f8620 msgr2=0x7f43a41013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:54:54.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:54.478+0000 7f43aae22700 1 -- 172.21.15.90:0/2464586981 shutdown_connections 2024-02-09T21:54:54.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:54:54.478+0000 7f43aae22700 1 -- 172.21.15.90:0/2464586981 wait complete. 2024-02-09T21:54:55.084 DEBUG:teuthology.orchestra.run.smithi090:osd.4> sudo journalctl -f -n 0 -u ceph-ee26d820-c794-11ee-95b8-87774f69a715@osd.4.service 2024-02-09T21:54:55.087 INFO:tasks.cephadm:Deploying osd.5 on smithi090 with /dev/vg_nvme/lv_2... 2024-02-09T21:54:55.087 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-09T21:54:55.390 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi090/config 2024-02-09T21:54:55.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:55 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:55.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:55 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:55.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:55 smithi090 ceph-mon[36996]: from='osd.4 [v2:172.21.15.90:6808/204678191,v1:172.21.15.90:6809/204678191]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-09T21:54:55.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:55 smithi090 ceph-mon[36996]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-09T21:54:55.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:55 smithi090 ceph-mon[36996]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-09T21:54:55.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:55 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:55.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:55 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:55.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:55 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:55.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:55 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:55.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:55 smithi006 ceph-mon[29258]: from='osd.4 [v2:172.21.15.90:6808/204678191,v1:172.21.15.90:6809/204678191]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-09T21:54:55.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:55 smithi006 ceph-mon[29258]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-09T21:54:55.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:55 smithi006 ceph-mon[29258]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-09T21:54:55.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:55 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:55.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:55 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:54:56.623 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:56 smithi090 ceph-mon[36996]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-09T21:54:56.623 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:56 smithi090 ceph-mon[36996]: osdmap e32: 5 total, 4 up, 5 in 2024-02-09T21:54:56.623 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:56 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T21:54:56.623 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:56 smithi090 ceph-mon[36996]: from='osd.4 [v2:172.21.15.90:6808/204678191,v1:172.21.15.90:6809/204678191]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-09T21:54:56.623 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:56 smithi090 ceph-mon[36996]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-09T21:54:56.623 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 09 21:54:56 smithi090 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-4[49210]: 2024-02-09T21:54:56.442+0000 7fafe65cf700 -1 osd.4 0 waiting for initial osdmap 2024-02-09T21:54:56.623 INFO:journalctl@ceph.osd.4.smithi090.stdout:Feb 09 21:54:56 smithi090 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-4[49210]: 2024-02-09T21:54:56.450+0000 7fafe33ce700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T21:54:56.884 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:56 smithi006 ceph-mon[29258]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-09T21:54:56.885 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:56 smithi006 ceph-mon[29258]: osdmap e32: 5 total, 4 up, 5 in 2024-02-09T21:54:56.885 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:56 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T21:54:56.885 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:56 smithi006 ceph-mon[29258]: from='osd.4 [v2:172.21.15.90:6808/204678191,v1:172.21.15.90:6809/204678191]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-09T21:54:56.885 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:56 smithi006 ceph-mon[29258]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-09T21:54:57.168 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-09T21:54:57.168 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-09T21:54:57.168 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-02-09T21:54:57.169 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-02-09T21:54:57.169 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0134386 s, 780 MB/s 2024-02-09T21:54:57.169 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-02-09T21:54:57.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:57 smithi090 ceph-mon[36996]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-02-09T21:54:57.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:57 smithi090 ceph-mon[36996]: osdmap e33: 5 total, 4 up, 5 in 2024-02-09T21:54:57.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:57 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T21:54:57.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:57 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T21:54:57.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:57 smithi090 ceph-mon[36996]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-09T21:54:57.706 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:57 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:54:57.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:57 smithi006 ceph-mon[29258]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-02-09T21:54:57.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:57 smithi006 ceph-mon[29258]: osdmap e33: 5 total, 4 up, 5 in 2024-02-09T21:54:57.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:57 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T21:54:57.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:57 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T21:54:57.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:57 smithi006 ceph-mon[29258]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-09T21:54:57.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:57 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:54:58.451 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi090:vg_nvme/lv_2 2024-02-09T21:54:58.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:58 smithi090 ceph-mon[36996]: purged_snaps scrub starts 2024-02-09T21:54:58.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:58 smithi090 ceph-mon[36996]: purged_snaps scrub ok 2024-02-09T21:54:58.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:58 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T21:54:58.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:58 smithi090 ceph-mon[36996]: osd.4 [v2:172.21.15.90:6808/204678191,v1:172.21.15.90:6809/204678191] boot 2024-02-09T21:54:58.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:58 smithi090 ceph-mon[36996]: osdmap e34: 5 total, 5 up, 5 in 2024-02-09T21:54:58.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:58 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T21:54:58.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:58 smithi006 ceph-mon[29258]: purged_snaps scrub starts 2024-02-09T21:54:58.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:58 smithi006 ceph-mon[29258]: purged_snaps scrub ok 2024-02-09T21:54:58.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:58 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T21:54:58.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:58 smithi006 ceph-mon[29258]: osd.4 [v2:172.21.15.90:6808/204678191,v1:172.21.15.90:6809/204678191] boot 2024-02-09T21:54:58.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:58 smithi006 ceph-mon[29258]: osdmap e34: 5 total, 5 up, 5 in 2024-02-09T21:54:58.908 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:58 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T21:54:59.170 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi090/config 2024-02-09T21:54:59.637 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:59 smithi090 ceph-mon[36996]: osdmap e35: 5 total, 5 up, 5 in 2024-02-09T21:54:59.637 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:54:59 smithi090 ceph-mon[36996]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-09T21:54:59.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:59 smithi006 ceph-mon[29258]: osdmap e35: 5 total, 5 up, 5 in 2024-02-09T21:54:59.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:54:59 smithi006 ceph-mon[29258]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-09T21:55:00.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.976+0000 7f658fff8700 1 -- 172.21.15.90:0/1782108238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6588066bd0 msgr2=0x7f65880ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:00.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.976+0000 7f658fff8700 1 --2- 172.21.15.90:0/1782108238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6588066bd0 0x7f65880ffba0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6584004940 tx=0x7f6584009e30 comp rx=0 tx=0).stop 2024-02-09T21:55:00.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.976+0000 7f658fff8700 1 -- 172.21.15.90:0/1782108238 shutdown_connections 2024-02-09T21:55:00.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.976+0000 7f658fff8700 1 --2- 172.21.15.90:0/1782108238 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65881000e0 0x7f65881024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:00.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.976+0000 7f658fff8700 1 --2- 172.21.15.90:0/1782108238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6588066bd0 0x7f65880ffba0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:00.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.977+0000 7f658fff8700 1 -- 172.21.15.90:0/1782108238 >> 172.21.15.90:0/1782108238 conn(0x7f65880f7590 msgr2=0x7f65880f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:55:00.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.977+0000 7f658fff8700 1 -- 172.21.15.90:0/1782108238 shutdown_connections 2024-02-09T21:55:00.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.977+0000 7f658fff8700 1 -- 172.21.15.90:0/1782108238 wait complete. 2024-02-09T21:55:00.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.977+0000 7f658fff8700 1 Processor -- start 2024-02-09T21:55:00.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.977+0000 7f658fff8700 1 -- start start 2024-02-09T21:55:00.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.977+0000 7f658fff8700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6588066bd0 0x7f658810d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:00.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.977+0000 7f658fff8700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65881000e0 0x7f658810df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:00.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.977+0000 7f658fff8700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658810e590 con 0x7f6588066bd0 2024-02-09T21:55:00.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.977+0000 7f658fff8700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658810edd0 con 0x7f65881000e0 2024-02-09T21:55:00.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.977+0000 7f658d593700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65881000e0 0x7f658810df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:00.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.977+0000 7f658d593700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65881000e0 0x7f658810df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52030/0 (socket says 172.21.15.90:52030) 2024-02-09T21:55:00.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.977+0000 7f658d593700 1 -- 172.21.15.90:0/1775498742 learned_addr learned my addr 172.21.15.90:0/1775498742 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-09T21:55:00.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.978+0000 7f658dd94700 1 --2- 172.21.15.90:0/1775498742 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6588066bd0 0x7f658810d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:00.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.978+0000 7f658d593700 1 -- 172.21.15.90:0/1775498742 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6588066bd0 msgr2=0x7f658810d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:00.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.978+0000 7f658d593700 1 --2- 172.21.15.90:0/1775498742 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6588066bd0 0x7f658810d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:00.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.978+0000 7f658d593700 1 -- 172.21.15.90:0/1775498742 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6584005040 con 0x7f65881000e0 2024-02-09T21:55:00.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.978+0000 7f658d593700 1 --2- 172.21.15.90:0/1775498742 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65881000e0 0x7f658810df00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f657800b790 tx=0x7f6578009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:00.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.978+0000 7f657e7fc700 1 -- 172.21.15.90:0/1775498742 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f657800f770 con 0x7f65881000e0 2024-02-09T21:55:00.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.978+0000 7f657e7fc700 1 -- 172.21.15.90:0/1775498742 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f657800fda0 con 0x7f65881000e0 2024-02-09T21:55:00.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.978+0000 7f658fff8700 1 -- 172.21.15.90:0/1775498742 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f658810f0b0 con 0x7f65881000e0 2024-02-09T21:55:00.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.978+0000 7f657e7fc700 1 -- 172.21.15.90:0/1775498742 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f657800d310 con 0x7f65881000e0 2024-02-09T21:55:00.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.978+0000 7f658fff8700 1 -- 172.21.15.90:0/1775498742 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f658810f5f0 con 0x7f65881000e0 2024-02-09T21:55:00.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.979+0000 7f658fff8700 1 -- 172.21.15.90:0/1775498742 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f658804bed0 con 0x7f65881000e0 2024-02-09T21:55:00.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.979+0000 7f657e7fc700 1 -- 172.21.15.90:0/1775498742 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f6578010420 con 0x7f65881000e0 2024-02-09T21:55:00.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.980+0000 7f657e7fc700 1 --2- 172.21.15.90:0/1775498742 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f65740692b0 0x7f657406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:00.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.980+0000 7f657e7fc700 1 -- 172.21.15.90:0/1775498742 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4202+0+0 (secure 0 0 0) 0x7f6578086bb0 con 0x7f65881000e0 2024-02-09T21:55:00.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.980+0000 7f658dd94700 1 --2- 172.21.15.90:0/1775498742 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f65740692b0 0x7f657406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:00.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.980+0000 7f658dd94700 1 --2- 172.21.15.90:0/1775498742 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f65740692b0 0x7f657406b760 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f658400b3b0 tx=0x7f6584009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:00.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:00.984+0000 7f657e7fc700 1 -- 172.21.15.90:0/1775498742 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f657808d080 con 0x7f65881000e0 2024-02-09T21:55:01.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:01.180+0000 7f658fff8700 1 -- 172.21.15.90:0/1775498742 --> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f658810fb70 con 0x7f65740692b0 2024-02-09T21:55:02.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:01 smithi006 ceph-mon[29258]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-09T21:55:02.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:01 smithi006 ceph-mon[29258]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:55:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T21:55:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T21:55:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:55:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-09T21:55:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-09T21:55:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:55:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:55:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:55:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:55:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:55:02.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:02.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:01 smithi090 ceph-mon[36996]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-09T21:55:02.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:01 smithi090 ceph-mon[36996]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:55:02.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T21:55:02.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T21:55:02.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:55:02.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:02.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:02.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-09T21:55:02.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-09T21:55:02.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:02.206 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:55:02.206 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:55:02.206 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:02.206 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:55:02.206 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:55:02.206 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:55:02.206 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:03.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:02 smithi090 ceph-mon[36996]: Detected new or changed devices on smithi090 2024-02-09T21:55:03.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:02 smithi090 ceph-mon[36996]: Adjusting osd_memory_target on smithi090 to 7968M 2024-02-09T21:55:03.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:02 smithi006 ceph-mon[29258]: Detected new or changed devices on smithi090 2024-02-09T21:55:03.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:02 smithi006 ceph-mon[29258]: Adjusting osd_memory_target on smithi090 to 7968M 2024-02-09T21:55:04.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:03 smithi090 ceph-mon[36996]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-09T21:55:04.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:03 smithi090 ceph-mon[36996]: from='client.? 172.21.15.90:0/1323975191' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "155a408e-ef3b-4c0a-8477-47fc6250b589"}]: dispatch 2024-02-09T21:55:04.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:03 smithi090 ceph-mon[36996]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "155a408e-ef3b-4c0a-8477-47fc6250b589"}]: dispatch 2024-02-09T21:55:04.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:03 smithi090 ceph-mon[36996]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "155a408e-ef3b-4c0a-8477-47fc6250b589"}]': finished 2024-02-09T21:55:04.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:03 smithi090 ceph-mon[36996]: osdmap e36: 6 total, 5 up, 6 in 2024-02-09T21:55:04.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:03 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T21:55:04.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:03 smithi006 ceph-mon[29258]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-09T21:55:04.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:03 smithi006 ceph-mon[29258]: from='client.? 172.21.15.90:0/1323975191' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "155a408e-ef3b-4c0a-8477-47fc6250b589"}]: dispatch 2024-02-09T21:55:04.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:03 smithi006 ceph-mon[29258]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "155a408e-ef3b-4c0a-8477-47fc6250b589"}]: dispatch 2024-02-09T21:55:04.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:03 smithi006 ceph-mon[29258]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "155a408e-ef3b-4c0a-8477-47fc6250b589"}]': finished 2024-02-09T21:55:04.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:03 smithi006 ceph-mon[29258]: osdmap e36: 6 total, 5 up, 6 in 2024-02-09T21:55:04.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:03 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T21:55:05.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:04 smithi090 ceph-mon[36996]: from='client.? 172.21.15.90:0/1004447274' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T21:55:05.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:04 smithi006 ceph-mon[29258]: from='client.? 172.21.15.90:0/1004447274' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T21:55:06.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:05 smithi090 ceph-mon[36996]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-09T21:55:06.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:05 smithi006 ceph-mon[29258]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-09T21:55:07.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:07 smithi006 ceph-mon[29258]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-09T21:55:07.704 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:07 smithi090 ceph-mon[36996]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-09T21:55:10.041 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:09 smithi090 ceph-mon[36996]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-09T21:55:10.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:09 smithi006 ceph-mon[29258]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-09T21:55:12.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:11 smithi090 ceph-mon[36996]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-09T21:55:12.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:11 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-09T21:55:12.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:11 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:55:12.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:11 smithi090 ceph-mon[36996]: Deploying daemon osd.5 on smithi090 2024-02-09T21:55:12.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:11 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:55:12.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:11 smithi006 ceph-mon[29258]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-09T21:55:12.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:11 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-09T21:55:12.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:11 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:55:12.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:11 smithi006 ceph-mon[29258]: Deploying daemon osd.5 on smithi090 2024-02-09T21:55:12.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:11 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:55:13.985 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:13 smithi090 ceph-mon[36996]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-09T21:55:14.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:13 smithi006 ceph-mon[29258]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-09T21:55:16.034 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:15 smithi090 ceph-mon[36996]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-09T21:55:16.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:15 smithi006 ceph-mon[29258]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-09T21:55:17.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:17 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:17.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:17 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:17.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:17 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:55:17.590 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:17 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:17.590 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:17 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:17.590 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:17 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:55:18.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:18 smithi090 ceph-mon[36996]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-09T21:55:18.545 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:18 smithi006 ceph-mon[29258]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-09T21:55:19.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:19.298+0000 7f657e7fc700 1 -- 172.21.15.90:0/1775498742 <== mgr.14216 v2:172.21.15.6:6800/2624137529 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f658810fb70 con 0x7f65740692b0 2024-02-09T21:55:19.302 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 5 on host 'smithi090' 2024-02-09T21:55:19.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:19.300+0000 7f658fff8700 1 -- 172.21.15.90:0/1775498742 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f65740692b0 msgr2=0x7f657406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:19.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:19.300+0000 7f658fff8700 1 --2- 172.21.15.90:0/1775498742 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f65740692b0 0x7f657406b760 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f658400b3b0 tx=0x7f6584009e30 comp rx=0 tx=0).stop 2024-02-09T21:55:19.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:19.300+0000 7f658fff8700 1 -- 172.21.15.90:0/1775498742 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65881000e0 msgr2=0x7f658810df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:19.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:19.300+0000 7f658fff8700 1 --2- 172.21.15.90:0/1775498742 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65881000e0 0x7f658810df00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f657800b790 tx=0x7f6578009250 comp rx=0 tx=0).stop 2024-02-09T21:55:19.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:19.300+0000 7f658fff8700 1 -- 172.21.15.90:0/1775498742 shutdown_connections 2024-02-09T21:55:19.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:19.300+0000 7f658fff8700 1 --2- 172.21.15.90:0/1775498742 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f65740692b0 0x7f657406b760 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:19.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:19.300+0000 7f658fff8700 1 --2- 172.21.15.90:0/1775498742 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6588066bd0 0x7f658810d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:19.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:19.300+0000 7f658fff8700 1 --2- 172.21.15.90:0/1775498742 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65881000e0 0x7f658810df00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:19.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:19.300+0000 7f658fff8700 1 -- 172.21.15.90:0/1775498742 >> 172.21.15.90:0/1775498742 conn(0x7f65880f7590 msgr2=0x7f65880f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:55:19.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:19.300+0000 7f658fff8700 1 -- 172.21.15.90:0/1775498742 shutdown_connections 2024-02-09T21:55:19.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:19.301+0000 7f658fff8700 1 -- 172.21.15.90:0/1775498742 wait complete. 2024-02-09T21:55:20.155 DEBUG:teuthology.orchestra.run.smithi090:osd.5> sudo journalctl -f -n 0 -u ceph-ee26d820-c794-11ee-95b8-87774f69a715@osd.5.service 2024-02-09T21:55:20.158 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-02-09T21:55:20.159 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph osd stat -f json 2024-02-09T21:55:20.435 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:55:20.454 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:20 smithi090 ceph-mon[36996]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-09T21:55:20.454 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:20 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:20.454 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:20 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:20.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:20 smithi090 ceph-mon[36996]: from='osd.5 [v2:172.21.15.90:6816/3296441764,v1:172.21.15.90:6817/3296441764]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-09T21:55:20.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:20 smithi090 ceph-mon[36996]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-09T21:55:20.556 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:20 smithi006 ceph-mon[29258]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-09T21:55:20.556 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:20 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:20.556 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:20 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:20.557 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:20 smithi006 ceph-mon[29258]: from='osd.5 [v2:172.21.15.90:6816/3296441764,v1:172.21.15.90:6817/3296441764]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-09T21:55:20.557 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:20 smithi006 ceph-mon[29258]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-09T21:55:21.550 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:21 smithi006 ceph-mon[29258]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-09T21:55:21.550 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:21 smithi006 ceph-mon[29258]: osdmap e37: 6 total, 5 up, 6 in 2024-02-09T21:55:21.551 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:21 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T21:55:21.551 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:21 smithi006 ceph-mon[29258]: from='osd.5 [v2:172.21.15.90:6816/3296441764,v1:172.21.15.90:6817/3296441764]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-09T21:55:21.551 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:21 smithi006 ceph-mon[29258]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-09T21:55:21.551 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:21 smithi006 ceph-mon[29258]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-09T21:55:21.551 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:21 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:21.551 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:21 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:21.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:21 smithi090 ceph-mon[36996]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-09T21:55:21.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:21 smithi090 ceph-mon[36996]: osdmap e37: 6 total, 5 up, 6 in 2024-02-09T21:55:21.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:21 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T21:55:21.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:21 smithi090 ceph-mon[36996]: from='osd.5 [v2:172.21.15.90:6816/3296441764,v1:172.21.15.90:6817/3296441764]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-09T21:55:21.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:21 smithi090 ceph-mon[36996]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-02-09T21:55:21.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:21 smithi090 ceph-mon[36996]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-09T21:55:21.706 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:21 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:21.706 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:21 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:21.706 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 09 21:55:21 smithi090 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-5[55677]: 2024-02-09T21:55:21.277+0000 7f591baa5700 -1 osd.5 0 waiting for initial osdmap 2024-02-09T21:55:21.706 INFO:journalctl@ceph.osd.5.smithi090.stdout:Feb 09 21:55:21 smithi090 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-5[55677]: 2024-02-09T21:55:21.288+0000 7f59168a0700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T21:55:22.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.253+0000 7f9365dd3700 1 -- 172.21.15.6:0/3503782121 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93600fbe40 msgr2=0x7f93600fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:22.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.253+0000 7f9365dd3700 1 --2- 172.21.15.6:0/3503782121 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93600fbe40 0x7f93600fc250 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9350004940 tx=0x7f9350009e30 comp rx=0 tx=0).stop 2024-02-09T21:55:22.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.253+0000 7f9365dd3700 1 -- 172.21.15.6:0/3503782121 shutdown_connections 2024-02-09T21:55:22.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.253+0000 7f9365dd3700 1 --2- 172.21.15.6:0/3503782121 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93600fcfb0 0x7f93600fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:22.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.253+0000 7f9365dd3700 1 --2- 172.21.15.6:0/3503782121 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93600fbe40 0x7f93600fc250 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:22.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.253+0000 7f9365dd3700 1 -- 172.21.15.6:0/3503782121 >> 172.21.15.6:0/3503782121 conn(0x7f93600f7520 msgr2=0x7f93600f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:55:22.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.253+0000 7f9365dd3700 1 -- 172.21.15.6:0/3503782121 shutdown_connections 2024-02-09T21:55:22.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.253+0000 7f9365dd3700 1 -- 172.21.15.6:0/3503782121 wait complete. 2024-02-09T21:55:22.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.253+0000 7f9365dd3700 1 Processor -- start 2024-02-09T21:55:22.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.253+0000 7f9365dd3700 1 -- start start 2024-02-09T21:55:22.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.254+0000 7f9365dd3700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93600fbe40 0x7f9360110650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:22.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.254+0000 7f9365dd3700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93600fcfb0 0x7f9360112ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:22.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.254+0000 7f9365dd3700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9360066cf0 con 0x7f93600fbe40 2024-02-09T21:55:22.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.254+0000 7f9365dd3700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9360066e30 con 0x7f93600fcfb0 2024-02-09T21:55:22.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.254+0000 7f935f7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93600fbe40 0x7f9360110650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:22.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.254+0000 7f935f7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93600fbe40 0x7f9360110650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60990/0 (socket says 172.21.15.6:60990) 2024-02-09T21:55:22.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.254+0000 7f935f7fe700 1 -- 172.21.15.6:0/322476125 learned_addr learned my addr 172.21.15.6:0/322476125 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:55:22.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.254+0000 7f935effd700 1 --2- 172.21.15.6:0/322476125 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93600fcfb0 0x7f9360112ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:22.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.254+0000 7f935f7fe700 1 -- 172.21.15.6:0/322476125 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93600fcfb0 msgr2=0x7f9360112ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:22.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.254+0000 7f935f7fe700 1 --2- 172.21.15.6:0/322476125 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93600fcfb0 0x7f9360112ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:22.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.254+0000 7f935f7fe700 1 -- 172.21.15.6:0/322476125 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9350005040 con 0x7f93600fbe40 2024-02-09T21:55:22.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.254+0000 7f935f7fe700 1 --2- 172.21.15.6:0/322476125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93600fbe40 0x7f9360110650 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f93500093b0 tx=0x7f9350009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:22.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.254+0000 7f9357fff700 1 -- 172.21.15.6:0/322476125 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f935000e040 con 0x7f93600fbe40 2024-02-09T21:55:22.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.255+0000 7f9365dd3700 1 -- 172.21.15.6:0/322476125 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93601131a0 con 0x7f93600fbe40 2024-02-09T21:55:22.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.255+0000 7f9357fff700 1 -- 172.21.15.6:0/322476125 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f935000bcc0 con 0x7f93600fbe40 2024-02-09T21:55:22.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.255+0000 7f9357fff700 1 -- 172.21.15.6:0/322476125 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f935001c630 con 0x7f93600fbe40 2024-02-09T21:55:22.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.255+0000 7f9365dd3700 1 -- 172.21.15.6:0/322476125 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93601136e0 con 0x7f93600fbe40 2024-02-09T21:55:22.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.256+0000 7f9365dd3700 1 -- 172.21.15.6:0/322476125 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f936004bed0 con 0x7f93600fbe40 2024-02-09T21:55:22.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.256+0000 7f9357fff700 1 -- 172.21.15.6:0/322476125 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f9350013980 con 0x7f93600fbe40 2024-02-09T21:55:22.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.257+0000 7f9357fff700 1 --2- 172.21.15.6:0/322476125 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f934c0692a0 0x7f934c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:22.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.257+0000 7f9357fff700 1 -- 172.21.15.6:0/322476125 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4354+0+0 (secure 0 0 0) 0x7f93500880d0 con 0x7f93600fbe40 2024-02-09T21:55:22.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.257+0000 7f935effd700 1 --2- 172.21.15.6:0/322476125 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f934c0692a0 0x7f934c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:22.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.257+0000 7f935effd700 1 --2- 172.21.15.6:0/322476125 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f934c0692a0 0x7f934c06b750 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9348004f40 tx=0x7f934800b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:22.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.261+0000 7f9357fff700 1 -- 172.21.15.6:0/322476125 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9350054850 con 0x7f93600fbe40 2024-02-09T21:55:22.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:22 smithi006 ceph-mon[29258]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-02-09T21:55:22.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:22 smithi006 ceph-mon[29258]: osdmap e38: 6 total, 5 up, 6 in 2024-02-09T21:55:22.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:22 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T21:55:22.408 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:22 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T21:55:22.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:22 smithi090 ceph-mon[36996]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-02-09T21:55:22.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:22 smithi090 ceph-mon[36996]: osdmap e38: 6 total, 5 up, 6 in 2024-02-09T21:55:22.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:22 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T21:55:22.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:22 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T21:55:22.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.454+0000 7f9365dd3700 1 -- 172.21.15.6:0/322476125 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f9360104930 con 0x7f93600fbe40 2024-02-09T21:55:22.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.454+0000 7f9357fff700 1 -- 172.21.15.6:0/322476125 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v39) v1 ==== 74+0+130 (secure 0 0 0) 0x7f9360104930 con 0x7f93600fbe40 2024-02-09T21:55:22.458 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:55:22.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.456+0000 7f9365dd3700 1 -- 172.21.15.6:0/322476125 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f934c0692a0 msgr2=0x7f934c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:22.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.456+0000 7f9365dd3700 1 --2- 172.21.15.6:0/322476125 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f934c0692a0 0x7f934c06b750 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9348004f40 tx=0x7f934800b040 comp rx=0 tx=0).stop 2024-02-09T21:55:22.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.456+0000 7f9365dd3700 1 -- 172.21.15.6:0/322476125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93600fbe40 msgr2=0x7f9360110650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:22.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.456+0000 7f9365dd3700 1 --2- 172.21.15.6:0/322476125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93600fbe40 0x7f9360110650 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f93500093b0 tx=0x7f9350009e30 comp rx=0 tx=0).stop 2024-02-09T21:55:22.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.456+0000 7f9365dd3700 1 -- 172.21.15.6:0/322476125 shutdown_connections 2024-02-09T21:55:22.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.456+0000 7f9365dd3700 1 --2- 172.21.15.6:0/322476125 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f934c0692a0 0x7f934c06b750 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:22.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.456+0000 7f9365dd3700 1 --2- 172.21.15.6:0/322476125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93600fbe40 0x7f9360110650 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:22.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.456+0000 7f9365dd3700 1 --2- 172.21.15.6:0/322476125 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93600fcfb0 0x7f9360112ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:22.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.456+0000 7f9365dd3700 1 -- 172.21.15.6:0/322476125 >> 172.21.15.6:0/322476125 conn(0x7f93600f7520 msgr2=0x7f9360100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:55:22.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.456+0000 7f9365dd3700 1 -- 172.21.15.6:0/322476125 shutdown_connections 2024-02-09T21:55:22.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:22.456+0000 7f9365dd3700 1 -- 172.21.15.6:0/322476125 wait complete. 2024-02-09T21:55:23.195 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":39,"num_osds":6,"num_up_osds":6,"osd_up_since":1707515722,"num_in_osds":6,"osd_in_since":1707515703,"num_remapped_pgs":0} 2024-02-09T21:55:23.196 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-09T21:55:23.461 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:55:23.499 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:23 smithi006 ceph-mon[29258]: purged_snaps scrub starts 2024-02-09T21:55:23.500 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:23 smithi006 ceph-mon[29258]: purged_snaps scrub ok 2024-02-09T21:55:23.500 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:23 smithi006 ceph-mon[29258]: osd.5 [v2:172.21.15.90:6816/3296441764,v1:172.21.15.90:6817/3296441764] boot 2024-02-09T21:55:23.500 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:23 smithi006 ceph-mon[29258]: osdmap e39: 6 total, 6 up, 6 in 2024-02-09T21:55:23.500 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:23 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T21:55:23.500 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:23 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/322476125' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-09T21:55:23.500 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:23 smithi006 ceph-mon[29258]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-09T21:55:23.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:23 smithi090 ceph-mon[36996]: purged_snaps scrub starts 2024-02-09T21:55:23.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:23 smithi090 ceph-mon[36996]: purged_snaps scrub ok 2024-02-09T21:55:23.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:23 smithi090 ceph-mon[36996]: osd.5 [v2:172.21.15.90:6816/3296441764,v1:172.21.15.90:6817/3296441764] boot 2024-02-09T21:55:23.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:23 smithi090 ceph-mon[36996]: osdmap e39: 6 total, 6 up, 6 in 2024-02-09T21:55:23.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:23 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T21:55:23.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:23 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/322476125' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-09T21:55:23.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:23 smithi090 ceph-mon[36996]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-09T21:55:24.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:24 smithi006 ceph-mon[29258]: osdmap e40: 6 total, 6 up, 6 in 2024-02-09T21:55:24.564 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:24 smithi090 ceph-mon[36996]: osdmap e40: 6 total, 6 up, 6 in 2024-02-09T21:55:25.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.107+0000 7f1250757700 1 -- 172.21.15.6:0/3535372946 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1248101e60 msgr2=0x7f1248102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:25.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.107+0000 7f1250757700 1 --2- 172.21.15.6:0/3535372946 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1248101e60 0x7f1248102230 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1238009ce0 tx=0x7f1238004d10 comp rx=0 tx=0).stop 2024-02-09T21:55:25.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.107+0000 7f1250757700 1 -- 172.21.15.6:0/3535372946 shutdown_connections 2024-02-09T21:55:25.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.107+0000 7f1250757700 1 --2- 172.21.15.6:0/3535372946 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1248102800 0x7f12480fbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:25.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.107+0000 7f1250757700 1 --2- 172.21.15.6:0/3535372946 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1248101e60 0x7f1248102230 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:25.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.107+0000 7f1250757700 1 -- 172.21.15.6:0/3535372946 >> 172.21.15.6:0/3535372946 conn(0x7f12480f7a10 msgr2=0x7f12480f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:55:25.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.107+0000 7f1250757700 1 -- 172.21.15.6:0/3535372946 shutdown_connections 2024-02-09T21:55:25.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.107+0000 7f1250757700 1 -- 172.21.15.6:0/3535372946 wait complete. 2024-02-09T21:55:25.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.108+0000 7f1250757700 1 Processor -- start 2024-02-09T21:55:25.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.108+0000 7f1250757700 1 -- start start 2024-02-09T21:55:25.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.108+0000 7f1250757700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1248101e60 0x7f1248112af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:25.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.108+0000 7f1250757700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1248102800 0x7f1248113030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:25.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.108+0000 7f1250757700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1248113710 con 0x7f1248102800 2024-02-09T21:55:25.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.108+0000 7f1250757700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124810c9b0 con 0x7f1248101e60 2024-02-09T21:55:25.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.108+0000 7f124dcf2700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1248102800 0x7f1248113030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:25.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.108+0000 7f124dcf2700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1248102800 0x7f1248113030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32778/0 (socket says 172.21.15.6:32778) 2024-02-09T21:55:25.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.108+0000 7f124dcf2700 1 -- 172.21.15.6:0/1974638110 learned_addr learned my addr 172.21.15.6:0/1974638110 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:55:25.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.108+0000 7f124e4f3700 1 --2- 172.21.15.6:0/1974638110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1248101e60 0x7f1248112af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:25.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.108+0000 7f124dcf2700 1 -- 172.21.15.6:0/1974638110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1248101e60 msgr2=0x7f1248112af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:25.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.108+0000 7f124dcf2700 1 --2- 172.21.15.6:0/1974638110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1248101e60 0x7f1248112af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:25.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.108+0000 7f124dcf2700 1 -- 172.21.15.6:0/1974638110 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1238009940 con 0x7f1248102800 2024-02-09T21:55:25.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.109+0000 7f124dcf2700 1 --2- 172.21.15.6:0/1974638110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1248102800 0x7f1248113030 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f123c00b790 tx=0x7f123c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:25.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.109+0000 7f1246ffd700 1 -- 172.21.15.6:0/1974638110 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f123c00f6f0 con 0x7f1248102800 2024-02-09T21:55:25.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.109+0000 7f1246ffd700 1 -- 172.21.15.6:0/1974638110 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f123c00fd20 con 0x7f1248102800 2024-02-09T21:55:25.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.109+0000 7f1250757700 1 -- 172.21.15.6:0/1974638110 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f124810cc90 con 0x7f1248102800 2024-02-09T21:55:25.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.109+0000 7f1246ffd700 1 -- 172.21.15.6:0/1974638110 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f123c0103a0 con 0x7f1248102800 2024-02-09T21:55:25.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.109+0000 7f1250757700 1 -- 172.21.15.6:0/1974638110 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f124810d1d0 con 0x7f1248102800 2024-02-09T21:55:25.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.110+0000 7f1250757700 1 -- 172.21.15.6:0/1974638110 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12480fb610 con 0x7f1248102800 2024-02-09T21:55:25.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.110+0000 7f1246ffd700 1 -- 172.21.15.6:0/1974638110 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f123c011400 con 0x7f1248102800 2024-02-09T21:55:25.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.110+0000 7f1246ffd700 1 --2- 172.21.15.6:0/1974638110 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f12300692a0 0x7f123006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:25.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.111+0000 7f1246ffd700 1 -- 172.21.15.6:0/1974638110 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f123c086c40 con 0x7f1248102800 2024-02-09T21:55:25.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.111+0000 7f124e4f3700 1 --2- 172.21.15.6:0/1974638110 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f12300692a0 0x7f123006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:25.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.111+0000 7f124e4f3700 1 --2- 172.21.15.6:0/1974638110 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f12300692a0 0x7f123006b750 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f123800b740 tx=0x7f1238004d10 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:25.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.115+0000 7f1246ffd700 1 -- 172.21.15.6:0/1974638110 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f123c056d40 con 0x7f1248102800 2024-02-09T21:55:25.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.307+0000 7f1250757700 1 -- 172.21.15.6:0/1974638110 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f124810e0d0 con 0x7f1248102800 2024-02-09T21:55:25.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.308+0000 7f1246ffd700 1 -- 172.21.15.6:0/1974638110 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+10995 (secure 0 0 0) 0x7f123c0568d0 con 0x7f1248102800 2024-02-09T21:55:25.312 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:55:25.313 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":40,"fsid":"ee26d820-c794-11ee-95b8-87774f69a715","created":"2024-02-09T21:49:39.138061+0000","modified":"2024-02-09T21:55:23.282158+0000","last_up_change":"2024-02-09T21:55:22.274974+0000","last_in_change":"2024-02-09T21:55:03.608607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-09T21:54:09.146714+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"c78f3c5b-14ab-40b7-bd39-52cb040b6532","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":4292395427},{"type":"v1","addr":"172.21.15.6:6803","nonce":4292395427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":4292395427},{"type":"v1","addr":"172.21.15.6:6805","nonce":4292395427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":4292395427},{"type":"v1","addr":"172.21.15.6:6809","nonce":4292395427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":4292395427},{"type":"v1","addr":"172.21.15.6:6807","nonce":4292395427}]},"public_addr":"172.21.15.6:6803/4292395427","cluster_addr":"172.21.15.6:6805/4292395427","heartbeat_back_addr":"172.21.15.6:6809/4292395427","heartbeat_front_addr":"172.21.15.6:6807/4292395427","state":["exists","up"]},{"osd":1,"uuid":"c2257408-632e-4e27-ae69-99046b12b405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3556298518},{"type":"v1","addr":"172.21.15.6:6811","nonce":3556298518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3556298518},{"type":"v1","addr":"172.21.15.6:6813","nonce":3556298518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":3556298518},{"type":"v1","addr":"172.21.15.6:6817","nonce":3556298518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":3556298518},{"type":"v1","addr":"172.21.15.6:6815","nonce":3556298518}]},"public_addr":"172.21.15.6:6811/3556298518","cluster_addr":"172.21.15.6:6813/3556298518","heartbeat_back_addr":"172.21.15.6:6817/3556298518","heartbeat_front_addr":"172.21.15.6:6815/3556298518","state":["exists","up"]},{"osd":2,"uuid":"e56e4fc0-614d-48e6-a2d6-758c34487540","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3201508057},{"type":"v1","addr":"172.21.15.6:6819","nonce":3201508057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3201508057},{"type":"v1","addr":"172.21.15.6:6821","nonce":3201508057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3201508057},{"type":"v1","addr":"172.21.15.6:6825","nonce":3201508057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3201508057},{"type":"v1","addr":"172.21.15.6:6823","nonce":3201508057}]},"public_addr":"172.21.15.6:6819/3201508057","cluster_addr":"172.21.15.6:6821/3201508057","heartbeat_back_addr":"172.21.15.6:6825/3201508057","heartbeat_front_addr":"172.21.15.6:6823/3201508057","state":["exists","up"]},{"osd":3,"uuid":"0b2a19ac-de1e-46f4-a3e0-5a8321672e4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":1692709589},{"type":"v1","addr":"172.21.15.90:6801","nonce":1692709589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1692709589},{"type":"v1","addr":"172.21.15.90:6803","nonce":1692709589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":1692709589},{"type":"v1","addr":"172.21.15.90:6807","nonce":1692709589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":1692709589},{"type":"v1","addr":"172.21.15.90:6805","nonce":1692709589}]},"public_addr":"172.21.15.90:6801/1692709589","cluster_addr":"172.21.15.90:6803/1692709589","heartbeat_back_addr":"172.21.15.90:6807/1692709589","heartbeat_front_addr":"172.21.15.90:6805/1692709589","state":["exists","up"]},{"osd":4,"uuid":"d79e2bb6-cebd-4e3c-8cec-5fac286838e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":204678191},{"type":"v1","addr":"172.21.15.90:6809","nonce":204678191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":204678191},{"type":"v1","addr":"172.21.15.90:6811","nonce":204678191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":204678191},{"type":"v1","addr":"172.21.15.90:6815","nonce":204678191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":204678191},{"type":"v1","addr":"172.21.15.90:6813","nonce":204678191}]},"public_addr":"172.21.15.90:6809/204678191","cluster_addr":"172.21.15.90:6811/204678191","heartbeat_back_addr":"172.21.15.90:6815/204678191","heartbeat_front_addr":"172.21.15.90:6813/204678191","state":["exists","up"]},{"osd":5,"uuid":"155a408e-ef3b-4c0a-8477-47fc6250b589","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3296441764},{"type":"v1","addr":"172.21.15.90:6817","nonce":3296441764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":3296441764},{"type":"v1","addr":"172.21.15.90:6819","nonce":3296441764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":3296441764},{"type":"v1","addr":"172.21.15.90:6823","nonce":3296441764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":3296441764},{"type":"v1","addr":"172.21.15.90:6821","nonce":3296441764}]},"public_addr":"172.21.15.90:6817/3296441764","cluster_addr":"172.21.15.90:6819/3296441764","heartbeat_back_addr":"172.21.15.90:6823/3296441764","heartbeat_front_addr":"172.21.15.90:6821/3296441764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T21:53:10.093731+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T21:53:39.528761+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T21:54:06.766402+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T21:54:30.531184+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T21:54:55.632482+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T21:55:21.026419+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:0/1297965797":"2024-02-10T21:51:56.813980+0000","172.21.15.6:0/2706037691":"2024-02-10T21:51:56.813980+0000","172.21.15.6:0/3672447393":"2024-02-10T21:51:56.813980+0000","172.21.15.6:6801/3321564517":"2024-02-10T21:51:56.813980+0000","172.21.15.6:0/1651082446":"2024-02-10T21:50:40.245871+0000","172.21.15.6:6800/1422219104":"2024-02-10T21:50:40.245871+0000","172.21.15.6:0/3369982019":"2024-02-10T21:50:07.987703+0000","172.21.15.6:6801/2325806844":"2024-02-10T21:50:07.987703+0000","172.21.15.6:6800/3321564517":"2024-02-10T21:51:56.813980+0000","172.21.15.6:0/147879205":"2024-02-10T21:50:07.987703+0000","172.21.15.6:6801/1422219104":"2024-02-10T21:50:40.245871+0000","172.21.15.6:0/2336225467":"2024-02-10T21:50:07.987703+0000","172.21.15.6:6800/2325806844":"2024-02-10T21:50:07.987703+0000","172.21.15.6:0/965195423":"2024-02-10T21:50:40.245871+0000","172.21.15.6:0/362750223":"2024-02-10T21:50:40.245871+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-09T21:55:25.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.310+0000 7f1250757700 1 -- 172.21.15.6:0/1974638110 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f12300692a0 msgr2=0x7f123006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:25.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.310+0000 7f1250757700 1 --2- 172.21.15.6:0/1974638110 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f12300692a0 0x7f123006b750 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f123800b740 tx=0x7f1238004d10 comp rx=0 tx=0).stop 2024-02-09T21:55:25.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.310+0000 7f1250757700 1 -- 172.21.15.6:0/1974638110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1248102800 msgr2=0x7f1248113030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:25.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.310+0000 7f1250757700 1 --2- 172.21.15.6:0/1974638110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1248102800 0x7f1248113030 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f123c00b790 tx=0x7f123c009220 comp rx=0 tx=0).stop 2024-02-09T21:55:25.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.310+0000 7f1250757700 1 -- 172.21.15.6:0/1974638110 shutdown_connections 2024-02-09T21:55:25.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.310+0000 7f1250757700 1 --2- 172.21.15.6:0/1974638110 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f12300692a0 0x7f123006b750 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:25.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.310+0000 7f1250757700 1 --2- 172.21.15.6:0/1974638110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1248101e60 0x7f1248112af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:25.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.310+0000 7f1250757700 1 --2- 172.21.15.6:0/1974638110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1248102800 0x7f1248113030 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:25.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.310+0000 7f1250757700 1 -- 172.21.15.6:0/1974638110 >> 172.21.15.6:0/1974638110 conn(0x7f12480f7a10 msgr2=0x7f12480f93c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:55:25.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.311+0000 7f1250757700 1 -- 172.21.15.6:0/1974638110 shutdown_connections 2024-02-09T21:55:25.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:25.311+0000 7f1250757700 1 -- 172.21.15.6:0/1974638110 wait complete. 2024-02-09T21:55:25.795 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:25 smithi006 ceph-mon[29258]: Detected new or changed devices on smithi090 2024-02-09T21:55:25.795 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:25 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:25.795 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:25 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:25.795 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:25 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-09T21:55:25.795 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:25 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-09T21:55:25.795 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:25 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-09T21:55:25.795 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:25 smithi006 ceph-mon[29258]: Adjusting osd_memory_target on smithi090 to 5312M 2024-02-09T21:55:25.795 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:25 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:25.796 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:25 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:55:25.796 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:25 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:55:25.796 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:25 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:25.796 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:25 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:55:25.796 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:25 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:55:25.796 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:25 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:55:25.796 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:25 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:25.796 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:25 smithi006 ceph-mon[29258]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:25.797 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:25 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/1974638110' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T21:55:25.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:25 smithi090 ceph-mon[36996]: Detected new or changed devices on smithi090 2024-02-09T21:55:25.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:25 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:25.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:25 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:25.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:25 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-09T21:55:25.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:25 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-09T21:55:25.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:25 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-09T21:55:25.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:25 smithi090 ceph-mon[36996]: Adjusting osd_memory_target on smithi090 to 5312M 2024-02-09T21:55:25.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:25 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:25.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:25 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:55:25.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:25 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:55:25.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:25 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:25.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:25 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:55:25.956 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:25 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:55:25.956 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:25 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:55:25.956 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:25 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:55:25.956 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:25 smithi090 ceph-mon[36996]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:25.956 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:25 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/1974638110' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T21:55:26.109 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-09T21:54:09.146714+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-02-09T21:55:26.110 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-02-09T21:55:26.373 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:55:27.506 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:27 smithi006 ceph-mon[29258]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:27.506 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:27 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:55:27.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:27 smithi090 ceph-mon[36996]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:27.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:27 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:55:28.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.027+0000 7f29ef037700 1 -- 172.21.15.6:0/2210679812 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29e81038e0 msgr2=0x7f29e80fcf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:28.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.027+0000 7f29ef037700 1 --2- 172.21.15.6:0/2210679812 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29e81038e0 0x7f29e80fcf50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f29d0004940 tx=0x7f29d0009e30 comp rx=0 tx=0).stop 2024-02-09T21:55:28.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.027+0000 7f29ef037700 1 -- 172.21.15.6:0/2210679812 shutdown_connections 2024-02-09T21:55:28.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.027+0000 7f29ef037700 1 --2- 172.21.15.6:0/2210679812 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29e81038e0 0x7f29e80fcf50 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:28.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.027+0000 7f29ef037700 1 --2- 172.21.15.6:0/2210679812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29e8102f40 0x7f29e8103310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:28.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.027+0000 7f29ef037700 1 -- 172.21.15.6:0/2210679812 >> 172.21.15.6:0/2210679812 conn(0x7f29e80f8af0 msgr2=0x7f29e80faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:55:28.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.027+0000 7f29ef037700 1 -- 172.21.15.6:0/2210679812 shutdown_connections 2024-02-09T21:55:28.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.027+0000 7f29ef037700 1 -- 172.21.15.6:0/2210679812 wait complete. 2024-02-09T21:55:28.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.028+0000 7f29ef037700 1 Processor -- start 2024-02-09T21:55:28.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.028+0000 7f29ef037700 1 -- start start 2024-02-09T21:55:28.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.028+0000 7f29ef037700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29e8102f40 0x7f29e810b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:28.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.028+0000 7f29ef037700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29e81038e0 0x7f29e810bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:28.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.028+0000 7f29ef037700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e810e950 con 0x7f29e81038e0 2024-02-09T21:55:28.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.028+0000 7f29ef037700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e810ea90 con 0x7f29e8102f40 2024-02-09T21:55:28.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.028+0000 7f29e7fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29e81038e0 0x7f29e810bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:28.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.028+0000 7f29e7fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29e81038e0 0x7f29e810bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49546/0 (socket says 172.21.15.6:49546) 2024-02-09T21:55:28.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.028+0000 7f29e7fff700 1 -- 172.21.15.6:0/2245173111 learned_addr learned my addr 172.21.15.6:0/2245173111 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:55:28.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.028+0000 7f29ecdd3700 1 --2- 172.21.15.6:0/2245173111 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29e8102f40 0x7f29e810b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:28.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.028+0000 7f29e7fff700 1 -- 172.21.15.6:0/2245173111 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29e8102f40 msgr2=0x7f29e810b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:28.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.028+0000 7f29e7fff700 1 --2- 172.21.15.6:0/2245173111 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29e8102f40 0x7f29e810b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:28.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.028+0000 7f29e7fff700 1 -- 172.21.15.6:0/2245173111 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d0005040 con 0x7f29e81038e0 2024-02-09T21:55:28.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.029+0000 7f29e7fff700 1 --2- 172.21.15.6:0/2245173111 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29e81038e0 0x7f29e810bc50 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f29d00090d0 tx=0x7f29d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:28.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.029+0000 7f29e57fa700 1 -- 172.21.15.6:0/2245173111 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29d000e040 con 0x7f29e81038e0 2024-02-09T21:55:28.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.029+0000 7f29ef037700 1 -- 172.21.15.6:0/2245173111 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e810c2a0 con 0x7f29e81038e0 2024-02-09T21:55:28.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.029+0000 7f29e57fa700 1 -- 172.21.15.6:0/2245173111 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f29d0009430 con 0x7f29e81038e0 2024-02-09T21:55:28.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.029+0000 7f29e57fa700 1 -- 172.21.15.6:0/2245173111 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29d001c3f0 con 0x7f29e81038e0 2024-02-09T21:55:28.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.029+0000 7f29ef037700 1 -- 172.21.15.6:0/2245173111 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e811e620 con 0x7f29e81038e0 2024-02-09T21:55:28.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.030+0000 7f29ef037700 1 -- 172.21.15.6:0/2245173111 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29e80fc6f0 con 0x7f29e81038e0 2024-02-09T21:55:28.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.030+0000 7f29e57fa700 1 -- 172.21.15.6:0/2245173111 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f29d000bbf0 con 0x7f29e81038e0 2024-02-09T21:55:28.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.031+0000 7f29e57fa700 1 --2- 172.21.15.6:0/2245173111 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f29d40692a0 0x7f29d406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:28.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.031+0000 7f29e57fa700 1 -- 172.21.15.6:0/2245173111 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f29d0088190 con 0x7f29e81038e0 2024-02-09T21:55:28.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.031+0000 7f29ecdd3700 1 --2- 172.21.15.6:0/2245173111 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f29d40692a0 0x7f29d406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:28.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.031+0000 7f29ecdd3700 1 --2- 172.21.15.6:0/2245173111 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f29d40692a0 0x7f29d406b750 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f29e0007970 tx=0x7f29e0009da0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:28.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.035+0000 7f29e57fa700 1 -- 172.21.15.6:0/2245173111 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f29d00547a0 con 0x7f29e81038e0 2024-02-09T21:55:28.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.226+0000 7f29ef037700 1 -- 172.21.15.6:0/2245173111 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f29e810cef0 con 0x7f29e81038e0 2024-02-09T21:55:28.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.226+0000 7f29e57fa700 1 -- 172.21.15.6:0/2245173111 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f29d0057dc0 con 0x7f29e81038e0 2024-02-09T21:55:28.230 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-02-09T21:55:28.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.228+0000 7f29ef037700 1 -- 172.21.15.6:0/2245173111 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f29d40692a0 msgr2=0x7f29d406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:28.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.228+0000 7f29ef037700 1 --2- 172.21.15.6:0/2245173111 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f29d40692a0 0x7f29d406b750 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f29e0007970 tx=0x7f29e0009da0 comp rx=0 tx=0).stop 2024-02-09T21:55:28.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.228+0000 7f29ef037700 1 -- 172.21.15.6:0/2245173111 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29e81038e0 msgr2=0x7f29e810bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:28.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.228+0000 7f29ef037700 1 --2- 172.21.15.6:0/2245173111 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29e81038e0 0x7f29e810bc50 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f29d00090d0 tx=0x7f29d0009e30 comp rx=0 tx=0).stop 2024-02-09T21:55:28.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.228+0000 7f29ef037700 1 -- 172.21.15.6:0/2245173111 shutdown_connections 2024-02-09T21:55:28.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.228+0000 7f29ef037700 1 --2- 172.21.15.6:0/2245173111 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f29d40692a0 0x7f29d406b750 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:28.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.228+0000 7f29ef037700 1 --2- 172.21.15.6:0/2245173111 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29e8102f40 0x7f29e810b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:28.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.228+0000 7f29ef037700 1 --2- 172.21.15.6:0/2245173111 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29e81038e0 0x7f29e810bc50 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:28.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.228+0000 7f29ef037700 1 -- 172.21.15.6:0/2245173111 >> 172.21.15.6:0/2245173111 conn(0x7f29e80f8af0 msgr2=0x7f29e80fa4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:55:28.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.228+0000 7f29ef037700 1 -- 172.21.15.6:0/2245173111 shutdown_connections 2024-02-09T21:55:28.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:28.228+0000 7f29ef037700 1 -- 172.21.15.6:0/2245173111 wait complete. 2024-02-09T21:55:28.572 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:28 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/2245173111' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T21:55:28.704 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:28 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/2245173111' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T21:55:28.883 INFO:tasks.cephadm:Setting up client nodes... 2024-02-09T21:55:28.884 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-09T21:55:29.148 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:55:29.576 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:29 smithi006 ceph-mon[29258]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:29.704 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:29 smithi090 ceph-mon[36996]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:30.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.800+0000 7f6c64e1f700 1 -- 172.21.15.6:0/2779747341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6c600fbe50 msgr2=0x7f6c600fc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:30.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.800+0000 7f6c64e1f700 1 --2- 172.21.15.6:0/2779747341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6c600fbe50 0x7f6c600fc260 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6c50004940 tx=0x7f6c50009e30 comp rx=0 tx=0).stop 2024-02-09T21:55:30.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.800+0000 7f6c64e1f700 1 -- 172.21.15.6:0/2779747341 shutdown_connections 2024-02-09T21:55:30.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.800+0000 7f6c64e1f700 1 --2- 172.21.15.6:0/2779747341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c600fcfc0 0x7f6c600fd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:30.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.800+0000 7f6c64e1f700 1 --2- 172.21.15.6:0/2779747341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6c600fbe50 0x7f6c600fc260 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:30.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.800+0000 7f6c64e1f700 1 -- 172.21.15.6:0/2779747341 >> 172.21.15.6:0/2779747341 conn(0x7f6c600f7510 msgr2=0x7f6c600f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:55:30.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.800+0000 7f6c64e1f700 1 -- 172.21.15.6:0/2779747341 shutdown_connections 2024-02-09T21:55:30.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.800+0000 7f6c64e1f700 1 -- 172.21.15.6:0/2779747341 wait complete. 2024-02-09T21:55:30.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.800+0000 7f6c64e1f700 1 Processor -- start 2024-02-09T21:55:30.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.800+0000 7f6c64e1f700 1 -- start start 2024-02-09T21:55:30.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.801+0000 7f6c64e1f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c600fbe50 0x7f6c6010c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:30.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.801+0000 7f6c64e1f700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6c600fcfc0 0x7f6c6010cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:30.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.801+0000 7f6c64e1f700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6010d390 con 0x7f6c600fcfc0 2024-02-09T21:55:30.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.801+0000 7f6c64e1f700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6010dc30 con 0x7f6c600fbe50 2024-02-09T21:55:30.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.801+0000 7f6c5dd9b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6c600fcfc0 0x7f6c6010cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:30.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.801+0000 7f6c5dd9b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6c600fcfc0 0x7f6c6010cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49574/0 (socket says 172.21.15.6:49574) 2024-02-09T21:55:30.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.801+0000 7f6c5dd9b700 1 -- 172.21.15.6:0/1896253981 learned_addr learned my addr 172.21.15.6:0/1896253981 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:55:30.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.801+0000 7f6c5e59c700 1 --2- 172.21.15.6:0/1896253981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c600fbe50 0x7f6c6010c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:30.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.801+0000 7f6c5dd9b700 1 -- 172.21.15.6:0/1896253981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c600fbe50 msgr2=0x7f6c6010c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:30.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.801+0000 7f6c5dd9b700 1 --2- 172.21.15.6:0/1896253981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c600fbe50 0x7f6c6010c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:30.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.801+0000 7f6c5dd9b700 1 -- 172.21.15.6:0/1896253981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c50005040 con 0x7f6c600fcfc0 2024-02-09T21:55:30.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.801+0000 7f6c5dd9b700 1 --2- 172.21.15.6:0/1896253981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6c600fcfc0 0x7f6c6010cd90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6c54002a50 tx=0x7f6c540046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:30.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.801+0000 7f6c4effd700 1 -- 172.21.15.6:0/1896253981 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c54007cc0 con 0x7f6c600fcfc0 2024-02-09T21:55:30.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.801+0000 7f6c4effd700 1 -- 172.21.15.6:0/1896253981 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c5400f040 con 0x7f6c600fcfc0 2024-02-09T21:55:30.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.801+0000 7f6c64e1f700 1 -- 172.21.15.6:0/1896253981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c6010df10 con 0x7f6c600fcfc0 2024-02-09T21:55:30.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.801+0000 7f6c4effd700 1 -- 172.21.15.6:0/1896253981 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c54009640 con 0x7f6c600fcfc0 2024-02-09T21:55:30.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.801+0000 7f6c64e1f700 1 -- 172.21.15.6:0/1896253981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c6010e450 con 0x7f6c600fcfc0 2024-02-09T21:55:30.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.802+0000 7f6c64e1f700 1 -- 172.21.15.6:0/1896253981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c6004bed0 con 0x7f6c600fcfc0 2024-02-09T21:55:30.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.803+0000 7f6c4effd700 1 -- 172.21.15.6:0/1896253981 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f6c540097a0 con 0x7f6c600fcfc0 2024-02-09T21:55:30.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.803+0000 7f6c4effd700 1 --2- 172.21.15.6:0/1896253981 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f6c480692b0 0x7f6c4806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:30.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.803+0000 7f6c4effd700 1 -- 172.21.15.6:0/1896253981 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6c54086b70 con 0x7f6c600fcfc0 2024-02-09T21:55:30.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.803+0000 7f6c5e59c700 1 --2- 172.21.15.6:0/1896253981 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f6c480692b0 0x7f6c4806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:30.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.804+0000 7f6c5e59c700 1 --2- 172.21.15.6:0/1896253981 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f6c480692b0 0x7f6c4806b760 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6c5000b3b0 tx=0x7f6c50009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:30.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:30.807+0000 7f6c4effd700 1 -- 172.21.15.6:0/1896253981 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6c54056c70 con 0x7f6c600fcfc0 2024-02-09T21:55:31.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:31.081+0000 7f6c64e1f700 1 -- 172.21.15.6:0/1896253981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f6c60104940 con 0x7f6c600fcfc0 2024-02-09T21:55:31.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:31.085+0000 7f6c4effd700 1 -- 172.21.15.6:0/1896253981 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7f6c54056800 con 0x7f6c600fcfc0 2024-02-09T21:55:31.089 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-02-09T21:55:31.089 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBTn8ZlwDLzBBAAZIxs7aqyYkSfDqPJq8peXQ== 2024-02-09T21:55:31.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:31.087+0000 7f6c64e1f700 1 -- 172.21.15.6:0/1896253981 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f6c480692b0 msgr2=0x7f6c4806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:31.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:31.087+0000 7f6c64e1f700 1 --2- 172.21.15.6:0/1896253981 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f6c480692b0 0x7f6c4806b760 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6c5000b3b0 tx=0x7f6c50009e30 comp rx=0 tx=0).stop 2024-02-09T21:55:31.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:31.087+0000 7f6c64e1f700 1 -- 172.21.15.6:0/1896253981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6c600fcfc0 msgr2=0x7f6c6010cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:31.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:31.087+0000 7f6c64e1f700 1 --2- 172.21.15.6:0/1896253981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6c600fcfc0 0x7f6c6010cd90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6c54002a50 tx=0x7f6c540046b0 comp rx=0 tx=0).stop 2024-02-09T21:55:31.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:31.087+0000 7f6c64e1f700 1 -- 172.21.15.6:0/1896253981 shutdown_connections 2024-02-09T21:55:31.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:31.087+0000 7f6c64e1f700 1 --2- 172.21.15.6:0/1896253981 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f6c480692b0 0x7f6c4806b760 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:31.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:31.087+0000 7f6c64e1f700 1 --2- 172.21.15.6:0/1896253981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c600fbe50 0x7f6c6010c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:31.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:31.087+0000 7f6c64e1f700 1 --2- 172.21.15.6:0/1896253981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6c600fcfc0 0x7f6c6010cd90 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:31.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:31.087+0000 7f6c64e1f700 1 -- 172.21.15.6:0/1896253981 >> 172.21.15.6:0/1896253981 conn(0x7f6c600f7510 msgr2=0x7f6c60100280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:55:31.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:31.087+0000 7f6c64e1f700 1 -- 172.21.15.6:0/1896253981 shutdown_connections 2024-02-09T21:55:31.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:31.087+0000 7f6c64e1f700 1 -- 172.21.15.6:0/1896253981 wait complete. 2024-02-09T21:55:31.901 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-02-09T21:55:31.901 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-09T21:55:31.901 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-09T21:55:31.947 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-09T21:55:32.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:31 smithi006 ceph-mon[29258]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:32.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:31 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/1896253981' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-09T21:55:32.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:31 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/1896253981' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-09T21:55:32.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:31 smithi090 ceph-mon[36996]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:32.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:31 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/1896253981' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-09T21:55:32.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:31 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/1896253981' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-09T21:55:32.212 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi090/config 2024-02-09T21:55:33.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.807+0000 7fe9c8769700 1 -- 172.21.15.90:0/1061094319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9c00fe0e0 msgr2=0x7fe9c00fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:33.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.807+0000 7fe9c8769700 1 --2- 172.21.15.90:0/1061094319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9c00fe0e0 0x7fe9c00fe550 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0004970 tx=0x7fe9b0009e30 comp rx=0 tx=0).stop 2024-02-09T21:55:33.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.807+0000 7fe9c8769700 1 -- 172.21.15.90:0/1061094319 shutdown_connections 2024-02-09T21:55:33.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.807+0000 7fe9c8769700 1 --2- 172.21.15.90:0/1061094319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9c00fe0e0 0x7fe9c00fe550 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:33.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.807+0000 7fe9c8769700 1 --2- 172.21.15.90:0/1061094319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9c00fcef0 0x7fe9c00fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:33.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.807+0000 7fe9c8769700 1 -- 172.21.15.90:0/1061094319 >> 172.21.15.90:0/1061094319 conn(0x7fe9c00f8620 msgr2=0x7fe9c00faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:55:33.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.807+0000 7fe9c8769700 1 -- 172.21.15.90:0/1061094319 shutdown_connections 2024-02-09T21:55:33.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.807+0000 7fe9c8769700 1 -- 172.21.15.90:0/1061094319 wait complete. 2024-02-09T21:55:33.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.808+0000 7fe9c8769700 1 Processor -- start 2024-02-09T21:55:33.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.808+0000 7fe9c8769700 1 -- start start 2024-02-09T21:55:33.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.808+0000 7fe9c8769700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9c00fcef0 0x7fe9c010d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:33.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.808+0000 7fe9c8769700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9c00fe0e0 0x7fe9c010de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:33.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.808+0000 7fe9c8769700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c010e430 con 0x7fe9c00fcef0 2024-02-09T21:55:33.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.808+0000 7fe9c8769700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c010ecd0 con 0x7fe9c00fe0e0 2024-02-09T21:55:33.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.808+0000 7fe9c5d04700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9c00fe0e0 0x7fe9c010de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:33.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.808+0000 7fe9c5d04700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9c00fe0e0 0x7fe9c010de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47772/0 (socket says 172.21.15.90:47772) 2024-02-09T21:55:33.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.808+0000 7fe9c5d04700 1 -- 172.21.15.90:0/973313754 learned_addr learned my addr 172.21.15.90:0/973313754 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-09T21:55:33.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.808+0000 7fe9c6505700 1 --2- 172.21.15.90:0/973313754 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9c00fcef0 0x7fe9c010d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:33.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.808+0000 7fe9c5d04700 1 -- 172.21.15.90:0/973313754 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9c00fcef0 msgr2=0x7fe9c010d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:33.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.808+0000 7fe9c5d04700 1 --2- 172.21.15.90:0/973313754 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9c00fcef0 0x7fe9c010d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:33.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.808+0000 7fe9c5d04700 1 -- 172.21.15.90:0/973313754 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b0005040 con 0x7fe9c00fe0e0 2024-02-09T21:55:33.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.808+0000 7fe9c5d04700 1 --2- 172.21.15.90:0/973313754 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9c00fe0e0 0x7fe9c010de30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0009360 tx=0x7fe9b0009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:33.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.808+0000 7fe9b6ffd700 1 -- 172.21.15.90:0/973313754 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9b000e040 con 0x7fe9c00fe0e0 2024-02-09T21:55:33.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.808+0000 7fe9c8769700 1 -- 172.21.15.90:0/973313754 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c010ef50 con 0x7fe9c00fe0e0 2024-02-09T21:55:33.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.808+0000 7fe9b6ffd700 1 -- 172.21.15.90:0/973313754 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe9b000bce0 con 0x7fe9c00fe0e0 2024-02-09T21:55:33.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.808+0000 7fe9b6ffd700 1 -- 172.21.15.90:0/973313754 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9b001c630 con 0x7fe9c00fe0e0 2024-02-09T21:55:33.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.809+0000 7fe9c8769700 1 -- 172.21.15.90:0/973313754 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c010f490 con 0x7fe9c00fe0e0 2024-02-09T21:55:33.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.810+0000 7fe9c8769700 1 -- 172.21.15.90:0/973313754 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c004bed0 con 0x7fe9c00fe0e0 2024-02-09T21:55:33.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.810+0000 7fe9b6ffd700 1 -- 172.21.15.90:0/973313754 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fe9b00139f0 con 0x7fe9c00fe0e0 2024-02-09T21:55:33.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.810+0000 7fe9b6ffd700 1 --2- 172.21.15.90:0/973313754 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fe9ac0692b0 0x7fe9ac06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:33.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.811+0000 7fe9b6ffd700 1 -- 172.21.15.90:0/973313754 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe9b00881c0 con 0x7fe9c00fe0e0 2024-02-09T21:55:33.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.811+0000 7fe9c6505700 1 --2- 172.21.15.90:0/973313754 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fe9ac0692b0 0x7fe9ac06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:33.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.811+0000 7fe9c6505700 1 --2- 172.21.15.90:0/973313754 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fe9ac0692b0 0x7fe9ac06b760 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe9c00fcc30 tx=0x7fe9bc009040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:33.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:33.815+0000 7fe9b6ffd700 1 -- 172.21.15.90:0/973313754 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe9b00582c0 con 0x7fe9c00fe0e0 2024-02-09T21:55:34.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:34.072+0000 7fe9c8769700 1 -- 172.21.15.90:0/973313754 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fe9c0105a30 con 0x7fe9c00fe0e0 2024-02-09T21:55:34.076 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:33 smithi090 ceph-mon[36996]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:34.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:34.079+0000 7fe9b6ffd700 1 -- 172.21.15.90:0/973313754 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7fe9b0057e50 con 0x7fe9c00fe0e0 2024-02-09T21:55:34.082 INFO:teuthology.orchestra.run.smithi090.stdout:[client.1] 2024-02-09T21:55:34.082 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQBWn8ZlX61rBBAA4wwABmxqu+J55P31G/KErA== 2024-02-09T21:55:34.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:34.081+0000 7fe9c8769700 1 -- 172.21.15.90:0/973313754 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fe9ac0692b0 msgr2=0x7fe9ac06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:34.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:34.081+0000 7fe9c8769700 1 --2- 172.21.15.90:0/973313754 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fe9ac0692b0 0x7fe9ac06b760 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe9c00fcc30 tx=0x7fe9bc009040 comp rx=0 tx=0).stop 2024-02-09T21:55:34.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:34.081+0000 7fe9c8769700 1 -- 172.21.15.90:0/973313754 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9c00fe0e0 msgr2=0x7fe9c010de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:34.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:34.081+0000 7fe9c8769700 1 --2- 172.21.15.90:0/973313754 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9c00fe0e0 0x7fe9c010de30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0009360 tx=0x7fe9b0009e30 comp rx=0 tx=0).stop 2024-02-09T21:55:34.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:34.081+0000 7fe9c8769700 1 -- 172.21.15.90:0/973313754 shutdown_connections 2024-02-09T21:55:34.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:34.081+0000 7fe9c8769700 1 --2- 172.21.15.90:0/973313754 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fe9ac0692b0 0x7fe9ac06b760 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:34.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:34.081+0000 7fe9c8769700 1 --2- 172.21.15.90:0/973313754 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9c00fcef0 0x7fe9c010d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:34.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:34.081+0000 7fe9c8769700 1 --2- 172.21.15.90:0/973313754 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9c00fe0e0 0x7fe9c010de30 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:34.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:34.081+0000 7fe9c8769700 1 -- 172.21.15.90:0/973313754 >> 172.21.15.90:0/973313754 conn(0x7fe9c00f8620 msgr2=0x7fe9c01013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:55:34.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:34.081+0000 7fe9c8769700 1 -- 172.21.15.90:0/973313754 shutdown_connections 2024-02-09T21:55:34.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-09T21:55:34.081+0000 7fe9c8769700 1 -- 172.21.15.90:0/973313754 wait complete. 2024-02-09T21:55:34.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:33 smithi006 ceph-mon[29258]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:34.734 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-09T21:55:34.735 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-09T21:55:34.735 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-09T21:55:34.779 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-09T21:55:34.779 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-09T21:55:34.779 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph mgr dump --format=json 2024-02-09T21:55:35.047 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:55:35.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:34 smithi006 ceph-mon[29258]: from='client.? 172.21.15.90:0/973313754' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-09T21:55:35.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:34 smithi006 ceph-mon[29258]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-09T21:55:35.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:34 smithi006 ceph-mon[29258]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-09T21:55:35.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:34 smithi090 ceph-mon[36996]: from='client.? 172.21.15.90:0/973313754' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-09T21:55:35.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:34 smithi090 ceph-mon[36996]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-09T21:55:35.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:34 smithi090 ceph-mon[36996]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-09T21:55:36.369 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:36 smithi006 ceph-mon[29258]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:36.454 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:36 smithi090 ceph-mon[36996]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:36.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.579+0000 7f0136eca700 1 -- 172.21.15.6:0/1796360186 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0130101e40 msgr2=0x7f0130102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:36.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.579+0000 7f0136eca700 1 --2- 172.21.15.6:0/1796360186 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0130101e40 0x7f0130102210 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0128009ce0 tx=0x7f0128004d10 comp rx=0 tx=0).stop 2024-02-09T21:55:36.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.579+0000 7f0136eca700 1 -- 172.21.15.6:0/1796360186 shutdown_connections 2024-02-09T21:55:36.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.579+0000 7f0136eca700 1 --2- 172.21.15.6:0/1796360186 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01301027e0 0x7f01300fbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:36.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.579+0000 7f0136eca700 1 --2- 172.21.15.6:0/1796360186 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0130101e40 0x7f0130102210 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:36.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.579+0000 7f0136eca700 1 -- 172.21.15.6:0/1796360186 >> 172.21.15.6:0/1796360186 conn(0x7f0130071d90 msgr2=0x7f0130072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:55:36.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.579+0000 7f0136eca700 1 -- 172.21.15.6:0/1796360186 shutdown_connections 2024-02-09T21:55:36.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.579+0000 7f0136eca700 1 -- 172.21.15.6:0/1796360186 wait complete. 2024-02-09T21:55:36.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.579+0000 7f0136eca700 1 Processor -- start 2024-02-09T21:55:36.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.579+0000 7f0136eca700 1 -- start start 2024-02-09T21:55:36.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.580+0000 7f0136eca700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0130101e40 0x7f013010ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:36.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.580+0000 7f0136eca700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01301027e0 0x7f013010f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:36.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.580+0000 7f0136eca700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013010f770 con 0x7f0130101e40 2024-02-09T21:55:36.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.580+0000 7f0136eca700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013010f8e0 con 0x7f01301027e0 2024-02-09T21:55:36.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.580+0000 7f0134c66700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0130101e40 0x7f013010ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:36.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.580+0000 7f0134c66700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0130101e40 0x7f013010ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49586/0 (socket says 172.21.15.6:49586) 2024-02-09T21:55:36.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.580+0000 7f0134c66700 1 -- 172.21.15.6:0/3456695277 learned_addr learned my addr 172.21.15.6:0/3456695277 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:55:36.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.580+0000 7f012ffff700 1 --2- 172.21.15.6:0/3456695277 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01301027e0 0x7f013010f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:36.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.580+0000 7f0134c66700 1 -- 172.21.15.6:0/3456695277 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01301027e0 msgr2=0x7f013010f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:36.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.580+0000 7f0134c66700 1 --2- 172.21.15.6:0/3456695277 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01301027e0 0x7f013010f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:36.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.580+0000 7f0134c66700 1 -- 172.21.15.6:0/3456695277 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0128009940 con 0x7f0130101e40 2024-02-09T21:55:36.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.580+0000 7f012ffff700 1 --2- 172.21.15.6:0/3456695277 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01301027e0 0x7f013010f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T21:55:36.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.580+0000 7f0134c66700 1 --2- 172.21.15.6:0/3456695277 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0130101e40 0x7f013010ec10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f012800bf90 tx=0x7f0128004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:36.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.580+0000 7f012d7fa700 1 -- 172.21.15.6:0/3456695277 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f012800e040 con 0x7f0130101e40 2024-02-09T21:55:36.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.580+0000 7f0136eca700 1 -- 172.21.15.6:0/3456695277 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0130117bd0 con 0x7f0130101e40 2024-02-09T21:55:36.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.580+0000 7f012d7fa700 1 -- 172.21.15.6:0/3456695277 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f01280128e0 con 0x7f0130101e40 2024-02-09T21:55:36.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.580+0000 7f012d7fa700 1 -- 172.21.15.6:0/3456695277 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f012801c8d0 con 0x7f0130101e40 2024-02-09T21:55:36.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.580+0000 7f0136eca700 1 -- 172.21.15.6:0/3456695277 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0130118110 con 0x7f0130101e40 2024-02-09T21:55:36.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.581+0000 7f0136eca700 1 -- 172.21.15.6:0/3456695277 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01300738b0 con 0x7f0130101e40 2024-02-09T21:55:36.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.582+0000 7f012d7fa700 1 -- 172.21.15.6:0/3456695277 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f0128012410 con 0x7f0130101e40 2024-02-09T21:55:36.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.582+0000 7f012d7fa700 1 --2- 172.21.15.6:0/3456695277 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f01180692a0 0x7f011806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:36.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.582+0000 7f012d7fa700 1 -- 172.21.15.6:0/3456695277 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0128088b00 con 0x7f0130101e40 2024-02-09T21:55:36.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.582+0000 7f012ffff700 1 --2- 172.21.15.6:0/3456695277 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f01180692a0 0x7f011806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:36.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.582+0000 7f012ffff700 1 --2- 172.21.15.6:0/3456695277 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f01180692a0 0x7f011806b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0124006fd0 tx=0x7f0124008040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:36.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.586+0000 7f012d7fa700 1 -- 172.21.15.6:0/3456695277 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f01280550f0 con 0x7f0130101e40 2024-02-09T21:55:36.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.831+0000 7f0136eca700 1 -- 172.21.15.6:0/3456695277 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f0130118300 con 0x7f0130101e40 2024-02-09T21:55:36.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.835+0000 7f012d7fa700 1 -- 172.21.15.6:0/3456695277 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+167158 (secure 0 0 0) 0x7f0128058710 con 0x7f0130101e40 2024-02-09T21:55:36.842 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:55:36.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.837+0000 7f0136eca700 1 -- 172.21.15.6:0/3456695277 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f01180692a0 msgr2=0x7f011806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:36.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.837+0000 7f0136eca700 1 --2- 172.21.15.6:0/3456695277 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f01180692a0 0x7f011806b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0124006fd0 tx=0x7f0124008040 comp rx=0 tx=0).stop 2024-02-09T21:55:36.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.837+0000 7f0136eca700 1 -- 172.21.15.6:0/3456695277 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0130101e40 msgr2=0x7f013010ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:36.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.837+0000 7f0136eca700 1 --2- 172.21.15.6:0/3456695277 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0130101e40 0x7f013010ec10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f012800bf90 tx=0x7f0128004d10 comp rx=0 tx=0).stop 2024-02-09T21:55:36.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.837+0000 7f0136eca700 1 -- 172.21.15.6:0/3456695277 shutdown_connections 2024-02-09T21:55:36.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.837+0000 7f0136eca700 1 --2- 172.21.15.6:0/3456695277 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f01180692a0 0x7f011806b750 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:36.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.837+0000 7f0136eca700 1 --2- 172.21.15.6:0/3456695277 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0130101e40 0x7f013010ec10 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:36.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.837+0000 7f0136eca700 1 --2- 172.21.15.6:0/3456695277 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01301027e0 0x7f013010f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:36.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.837+0000 7f0136eca700 1 -- 172.21.15.6:0/3456695277 >> 172.21.15.6:0/3456695277 conn(0x7f0130071d90 msgr2=0x7f01300fad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:55:36.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.837+0000 7f0136eca700 1 -- 172.21.15.6:0/3456695277 shutdown_connections 2024-02-09T21:55:36.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:36.837+0000 7f0136eca700 1 -- 172.21.15.6:0/3456695277 wait complete. 2024-02-09T21:55:37.485 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:37 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/3456695277' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-09T21:55:37.485 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:37 smithi006 ceph-mon[29258]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:37.511 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":19,"active_gid":14216,"active_name":"smithi006.glirxt","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2624137529},{"type":"v1","addr":"172.21.15.6:6801","nonce":2624137529}]},"active_addr":"172.21.15.6:6801/2624137529","active_change":"2024-02-09T21:51:56.814088+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14236,"name":"smithi090.csxosj","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.6:8443/","prometheus":"http://172.21.15.6:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":314251806}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":3970891558}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":8694484}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":3422519444}]}]} 2024-02-09T21:55:37.515 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-09T21:55:37.515 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-09T21:55:37.515 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-09T21:55:37.704 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:37 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/3456695277' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-09T21:55:37.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:37 smithi090 ceph-mon[36996]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:37.776 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:55:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.315+0000 7f5a1eec8700 1 -- 172.21.15.6:0/2932989591 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a18102f50 msgr2=0x7f5a18103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.315+0000 7f5a1eec8700 1 --2- 172.21.15.6:0/2932989591 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a18102f50 0x7f5a18103320 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5a00004940 tx=0x7f5a00009e30 comp rx=0 tx=0).stop 2024-02-09T21:55:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.315+0000 7f5a1eec8700 1 -- 172.21.15.6:0/2932989591 shutdown_connections 2024-02-09T21:55:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.315+0000 7f5a1eec8700 1 --2- 172.21.15.6:0/2932989591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a181038f0 0x7f5a180fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.315+0000 7f5a1eec8700 1 --2- 172.21.15.6:0/2932989591 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a18102f50 0x7f5a18103320 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.315+0000 7f5a1eec8700 1 -- 172.21.15.6:0/2932989591 >> 172.21.15.6:0/2932989591 conn(0x7f5a180f8ae0 msgr2=0x7f5a180faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:55:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.315+0000 7f5a1eec8700 1 -- 172.21.15.6:0/2932989591 shutdown_connections 2024-02-09T21:55:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.316+0000 7f5a1eec8700 1 -- 172.21.15.6:0/2932989591 wait complete. 2024-02-09T21:55:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.316+0000 7f5a1eec8700 1 Processor -- start 2024-02-09T21:55:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.316+0000 7f5a1eec8700 1 -- start start 2024-02-09T21:55:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.316+0000 7f5a1eec8700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a18102f50 0x7f5a18113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.316+0000 7f5a1eec8700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a181038f0 0x7f5a18113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.316+0000 7f5a1eec8700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a181145f0 con 0x7f5a181038f0 2024-02-09T21:55:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.316+0000 7f5a1eec8700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1810d920 con 0x7f5a18102f50 2024-02-09T21:55:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.317+0000 7f5a17fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a181038f0 0x7f5a18113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.317+0000 7f5a17fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a181038f0 0x7f5a18113fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33094/0 (socket says 172.21.15.6:33094) 2024-02-09T21:55:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.317+0000 7f5a17fff700 1 -- 172.21.15.6:0/1841233476 learned_addr learned my addr 172.21.15.6:0/1841233476 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:55:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.317+0000 7f5a1cc64700 1 --2- 172.21.15.6:0/1841233476 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a18102f50 0x7f5a18113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.317+0000 7f5a17fff700 1 -- 172.21.15.6:0/1841233476 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a18102f50 msgr2=0x7f5a18113a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.317+0000 7f5a17fff700 1 --2- 172.21.15.6:0/1841233476 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a18102f50 0x7f5a18113a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.317+0000 7f5a17fff700 1 -- 172.21.15.6:0/1841233476 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a00005040 con 0x7f5a181038f0 2024-02-09T21:55:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.317+0000 7f5a17fff700 1 --2- 172.21.15.6:0/1841233476 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a181038f0 0x7f5a18113fa0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5a0800c890 tx=0x7f5a080044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.317+0000 7f5a157fa700 1 -- 172.21.15.6:0/1841233476 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a0800a700 con 0x7f5a181038f0 2024-02-09T21:55:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.317+0000 7f5a157fa700 1 -- 172.21.15.6:0/1841233476 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a0800ad30 con 0x7f5a181038f0 2024-02-09T21:55:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.317+0000 7f5a1eec8700 1 -- 172.21.15.6:0/1841233476 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a1810dc00 con 0x7f5a181038f0 2024-02-09T21:55:39.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.317+0000 7f5a157fa700 1 -- 172.21.15.6:0/1841233476 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a080113a0 con 0x7f5a181038f0 2024-02-09T21:55:39.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.317+0000 7f5a1eec8700 1 -- 172.21.15.6:0/1841233476 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a1810e140 con 0x7f5a181038f0 2024-02-09T21:55:39.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.318+0000 7f5a1eec8700 1 -- 172.21.15.6:0/1841233476 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a18100260 con 0x7f5a181038f0 2024-02-09T21:55:39.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.319+0000 7f5a157fa700 1 -- 172.21.15.6:0/1841233476 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f5a08012400 con 0x7f5a181038f0 2024-02-09T21:55:39.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.319+0000 7f5a157fa700 1 --2- 172.21.15.6:0/1841233476 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f5a040692a0 0x7f5a0406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:39.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.319+0000 7f5a157fa700 1 -- 172.21.15.6:0/1841233476 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5a08086e00 con 0x7f5a181038f0 2024-02-09T21:55:39.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.319+0000 7f5a1cc64700 1 --2- 172.21.15.6:0/1841233476 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f5a040692a0 0x7f5a0406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:39.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.320+0000 7f5a1cc64700 1 --2- 172.21.15.6:0/1841233476 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f5a040692a0 0x7f5a0406b750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5a00004f90 tx=0x7f5a00009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:39.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.323+0000 7f5a157fa700 1 -- 172.21.15.6:0/1841233476 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f5a08056fa0 con 0x7f5a181038f0 2024-02-09T21:55:39.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.516+0000 7f5a1eec8700 1 -- 172.21.15.6:0/1841233476 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a1810a050 con 0x7f5a181038f0 2024-02-09T21:55:39.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.517+0000 7f5a157fa700 1 -- 172.21.15.6:0/1841233476 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+10995 (secure 0 0 0) 0x7f5a08056b30 con 0x7f5a181038f0 2024-02-09T21:55:39.522 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:55:39.522 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":40,"fsid":"ee26d820-c794-11ee-95b8-87774f69a715","created":"2024-02-09T21:49:39.138061+0000","modified":"2024-02-09T21:55:23.282158+0000","last_up_change":"2024-02-09T21:55:22.274974+0000","last_in_change":"2024-02-09T21:55:03.608607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-09T21:54:09.146714+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"c78f3c5b-14ab-40b7-bd39-52cb040b6532","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":4292395427},{"type":"v1","addr":"172.21.15.6:6803","nonce":4292395427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":4292395427},{"type":"v1","addr":"172.21.15.6:6805","nonce":4292395427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":4292395427},{"type":"v1","addr":"172.21.15.6:6809","nonce":4292395427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":4292395427},{"type":"v1","addr":"172.21.15.6:6807","nonce":4292395427}]},"public_addr":"172.21.15.6:6803/4292395427","cluster_addr":"172.21.15.6:6805/4292395427","heartbeat_back_addr":"172.21.15.6:6809/4292395427","heartbeat_front_addr":"172.21.15.6:6807/4292395427","state":["exists","up"]},{"osd":1,"uuid":"c2257408-632e-4e27-ae69-99046b12b405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3556298518},{"type":"v1","addr":"172.21.15.6:6811","nonce":3556298518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3556298518},{"type":"v1","addr":"172.21.15.6:6813","nonce":3556298518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":3556298518},{"type":"v1","addr":"172.21.15.6:6817","nonce":3556298518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":3556298518},{"type":"v1","addr":"172.21.15.6:6815","nonce":3556298518}]},"public_addr":"172.21.15.6:6811/3556298518","cluster_addr":"172.21.15.6:6813/3556298518","heartbeat_back_addr":"172.21.15.6:6817/3556298518","heartbeat_front_addr":"172.21.15.6:6815/3556298518","state":["exists","up"]},{"osd":2,"uuid":"e56e4fc0-614d-48e6-a2d6-758c34487540","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3201508057},{"type":"v1","addr":"172.21.15.6:6819","nonce":3201508057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3201508057},{"type":"v1","addr":"172.21.15.6:6821","nonce":3201508057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3201508057},{"type":"v1","addr":"172.21.15.6:6825","nonce":3201508057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3201508057},{"type":"v1","addr":"172.21.15.6:6823","nonce":3201508057}]},"public_addr":"172.21.15.6:6819/3201508057","cluster_addr":"172.21.15.6:6821/3201508057","heartbeat_back_addr":"172.21.15.6:6825/3201508057","heartbeat_front_addr":"172.21.15.6:6823/3201508057","state":["exists","up"]},{"osd":3,"uuid":"0b2a19ac-de1e-46f4-a3e0-5a8321672e4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":1692709589},{"type":"v1","addr":"172.21.15.90:6801","nonce":1692709589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1692709589},{"type":"v1","addr":"172.21.15.90:6803","nonce":1692709589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":1692709589},{"type":"v1","addr":"172.21.15.90:6807","nonce":1692709589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":1692709589},{"type":"v1","addr":"172.21.15.90:6805","nonce":1692709589}]},"public_addr":"172.21.15.90:6801/1692709589","cluster_addr":"172.21.15.90:6803/1692709589","heartbeat_back_addr":"172.21.15.90:6807/1692709589","heartbeat_front_addr":"172.21.15.90:6805/1692709589","state":["exists","up"]},{"osd":4,"uuid":"d79e2bb6-cebd-4e3c-8cec-5fac286838e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":204678191},{"type":"v1","addr":"172.21.15.90:6809","nonce":204678191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":204678191},{"type":"v1","addr":"172.21.15.90:6811","nonce":204678191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":204678191},{"type":"v1","addr":"172.21.15.90:6815","nonce":204678191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":204678191},{"type":"v1","addr":"172.21.15.90:6813","nonce":204678191}]},"public_addr":"172.21.15.90:6809/204678191","cluster_addr":"172.21.15.90:6811/204678191","heartbeat_back_addr":"172.21.15.90:6815/204678191","heartbeat_front_addr":"172.21.15.90:6813/204678191","state":["exists","up"]},{"osd":5,"uuid":"155a408e-ef3b-4c0a-8477-47fc6250b589","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3296441764},{"type":"v1","addr":"172.21.15.90:6817","nonce":3296441764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":3296441764},{"type":"v1","addr":"172.21.15.90:6819","nonce":3296441764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":3296441764},{"type":"v1","addr":"172.21.15.90:6823","nonce":3296441764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":3296441764},{"type":"v1","addr":"172.21.15.90:6821","nonce":3296441764}]},"public_addr":"172.21.15.90:6817/3296441764","cluster_addr":"172.21.15.90:6819/3296441764","heartbeat_back_addr":"172.21.15.90:6823/3296441764","heartbeat_front_addr":"172.21.15.90:6821/3296441764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T21:53:10.093731+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T21:53:39.528761+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T21:54:06.766402+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T21:54:30.531184+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T21:54:55.632482+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T21:55:21.026419+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:0/1297965797":"2024-02-10T21:51:56.813980+0000","172.21.15.6:0/2706037691":"2024-02-10T21:51:56.813980+0000","172.21.15.6:0/3672447393":"2024-02-10T21:51:56.813980+0000","172.21.15.6:6801/3321564517":"2024-02-10T21:51:56.813980+0000","172.21.15.6:0/1651082446":"2024-02-10T21:50:40.245871+0000","172.21.15.6:6800/1422219104":"2024-02-10T21:50:40.245871+0000","172.21.15.6:0/3369982019":"2024-02-10T21:50:07.987703+0000","172.21.15.6:6801/2325806844":"2024-02-10T21:50:07.987703+0000","172.21.15.6:6800/3321564517":"2024-02-10T21:51:56.813980+0000","172.21.15.6:0/147879205":"2024-02-10T21:50:07.987703+0000","172.21.15.6:6801/1422219104":"2024-02-10T21:50:40.245871+0000","172.21.15.6:0/2336225467":"2024-02-10T21:50:07.987703+0000","172.21.15.6:6800/2325806844":"2024-02-10T21:50:07.987703+0000","172.21.15.6:0/965195423":"2024-02-10T21:50:40.245871+0000","172.21.15.6:0/362750223":"2024-02-10T21:50:40.245871+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-09T21:55:39.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.519+0000 7f5a1eec8700 1 -- 172.21.15.6:0/1841233476 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f5a040692a0 msgr2=0x7f5a0406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:39.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.519+0000 7f5a1eec8700 1 --2- 172.21.15.6:0/1841233476 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f5a040692a0 0x7f5a0406b750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5a00004f90 tx=0x7f5a00009e30 comp rx=0 tx=0).stop 2024-02-09T21:55:39.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.519+0000 7f5a1eec8700 1 -- 172.21.15.6:0/1841233476 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a181038f0 msgr2=0x7f5a18113fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:39.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.519+0000 7f5a1eec8700 1 --2- 172.21.15.6:0/1841233476 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a181038f0 0x7f5a18113fa0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5a0800c890 tx=0x7f5a080044c0 comp rx=0 tx=0).stop 2024-02-09T21:55:39.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.519+0000 7f5a1eec8700 1 -- 172.21.15.6:0/1841233476 shutdown_connections 2024-02-09T21:55:39.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.519+0000 7f5a1eec8700 1 --2- 172.21.15.6:0/1841233476 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f5a040692a0 0x7f5a0406b750 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:39.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.519+0000 7f5a1eec8700 1 --2- 172.21.15.6:0/1841233476 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a18102f50 0x7f5a18113a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:39.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.519+0000 7f5a1eec8700 1 --2- 172.21.15.6:0/1841233476 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a181038f0 0x7f5a18113fa0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:39.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.519+0000 7f5a1eec8700 1 -- 172.21.15.6:0/1841233476 >> 172.21.15.6:0/1841233476 conn(0x7f5a180f8ae0 msgr2=0x7f5a180fa4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:55:39.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.519+0000 7f5a1eec8700 1 -- 172.21.15.6:0/1841233476 shutdown_connections 2024-02-09T21:55:39.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:39.519+0000 7f5a1eec8700 1 -- 172.21.15.6:0/1841233476 wait complete. 2024-02-09T21:55:40.156 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:39 smithi006 ceph-mon[29258]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:40.156 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:39 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/1841233476' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T21:55:40.183 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-09T21:55:40.184 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-09T21:55:40.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:39 smithi090 ceph-mon[36996]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:40.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:39 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/1841233476' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T21:55:40.446 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:55:42.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.155+0000 7f91702ee700 1 -- 172.21.15.6:0/3149439001 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91681027e0 msgr2=0x7f91680fbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:42.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.155+0000 7f91702ee700 1 --2- 172.21.15.6:0/3149439001 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91681027e0 0x7f91680fbea0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f915c004940 tx=0x7f915c009e30 comp rx=0 tx=0).stop 2024-02-09T21:55:42.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.155+0000 7f91702ee700 1 -- 172.21.15.6:0/3149439001 shutdown_connections 2024-02-09T21:55:42.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.155+0000 7f91702ee700 1 --2- 172.21.15.6:0/3149439001 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91681027e0 0x7f91680fbea0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:42.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.155+0000 7f91702ee700 1 --2- 172.21.15.6:0/3149439001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9168101e40 0x7f9168102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:42.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.155+0000 7f91702ee700 1 -- 172.21.15.6:0/3149439001 >> 172.21.15.6:0/3149439001 conn(0x7f9168071d90 msgr2=0x7f9168072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:55:42.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.155+0000 7f91702ee700 1 -- 172.21.15.6:0/3149439001 shutdown_connections 2024-02-09T21:55:42.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.155+0000 7f91702ee700 1 -- 172.21.15.6:0/3149439001 wait complete. 2024-02-09T21:55:42.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.155+0000 7f91702ee700 1 Processor -- start 2024-02-09T21:55:42.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.156+0000 7f91702ee700 1 -- start start 2024-02-09T21:55:42.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.156+0000 7f91702ee700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9168101e40 0x7f9168113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:42.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.156+0000 7f91702ee700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91681027e0 0x7f9168113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:42.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.156+0000 7f91702ee700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9168114600 con 0x7f9168101e40 2024-02-09T21:55:42.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.156+0000 7f91702ee700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916810d930 con 0x7f91681027e0 2024-02-09T21:55:42.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.156+0000 7f916e08a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9168101e40 0x7f9168113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:42.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.156+0000 7f916e08a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9168101e40 0x7f9168113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33116/0 (socket says 172.21.15.6:33116) 2024-02-09T21:55:42.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.156+0000 7f916e08a700 1 -- 172.21.15.6:0/831742638 learned_addr learned my addr 172.21.15.6:0/831742638 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:55:42.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.156+0000 7f916d889700 1 --2- 172.21.15.6:0/831742638 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91681027e0 0x7f9168113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:42.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.156+0000 7f916e08a700 1 -- 172.21.15.6:0/831742638 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91681027e0 msgr2=0x7f9168113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:42.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.156+0000 7f916e08a700 1 --2- 172.21.15.6:0/831742638 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91681027e0 0x7f9168113fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:42.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.156+0000 7f916e08a700 1 -- 172.21.15.6:0/831742638 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f915c005040 con 0x7f9168101e40 2024-02-09T21:55:42.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.156+0000 7f916d889700 1 --2- 172.21.15.6:0/831742638 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91681027e0 0x7f9168113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T21:55:42.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.156+0000 7f916e08a700 1 --2- 172.21.15.6:0/831742638 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9168101e40 0x7f9168113a70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f915800cb10 tx=0x7f9158007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:42.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.156+0000 7f9166ffd700 1 -- 172.21.15.6:0/831742638 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9158004200 con 0x7f9168101e40 2024-02-09T21:55:42.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.156+0000 7f9166ffd700 1 -- 172.21.15.6:0/831742638 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f915800acc0 con 0x7f9168101e40 2024-02-09T21:55:42.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.156+0000 7f91702ee700 1 -- 172.21.15.6:0/831742638 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f916810dc10 con 0x7f9168101e40 2024-02-09T21:55:42.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.156+0000 7f9166ffd700 1 -- 172.21.15.6:0/831742638 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9158003d90 con 0x7f9168101e40 2024-02-09T21:55:42.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.157+0000 7f91702ee700 1 -- 172.21.15.6:0/831742638 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f916810e150 con 0x7f9168101e40 2024-02-09T21:55:42.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.158+0000 7f91702ee700 1 -- 172.21.15.6:0/831742638 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91680738b0 con 0x7f9168101e40 2024-02-09T21:55:42.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.158+0000 7f9166ffd700 1 -- 172.21.15.6:0/831742638 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f9158003430 con 0x7f9168101e40 2024-02-09T21:55:42.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.158+0000 7f9166ffd700 1 --2- 172.21.15.6:0/831742638 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f91500692a0 0x7f915006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:42.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.159+0000 7f9166ffd700 1 -- 172.21.15.6:0/831742638 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f91580873e0 con 0x7f9168101e40 2024-02-09T21:55:42.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.159+0000 7f916d889700 1 --2- 172.21.15.6:0/831742638 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f91500692a0 0x7f915006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:42.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.159+0000 7f916d889700 1 --2- 172.21.15.6:0/831742638 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f91500692a0 0x7f915006b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f915c00b3b0 tx=0x7f915c009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:42.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.163+0000 7f9166ffd700 1 -- 172.21.15.6:0/831742638 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f91580574e0 con 0x7f9168101e40 2024-02-09T21:55:42.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:41 smithi006 ceph-mon[29258]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:42.167 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:41 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:55:42.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:41 smithi090 ceph-mon[36996]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:42.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:41 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:55:42.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.356+0000 7f91702ee700 1 -- 172.21.15.6:0/831742638 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f916810f050 con 0x7f9168101e40 2024-02-09T21:55:42.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.356+0000 7f9166ffd700 1 -- 172.21.15.6:0/831742638 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+10995 (secure 0 0 0) 0x7f9158057070 con 0x7f9168101e40 2024-02-09T21:55:42.363 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:55:42.363 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":40,"fsid":"ee26d820-c794-11ee-95b8-87774f69a715","created":"2024-02-09T21:49:39.138061+0000","modified":"2024-02-09T21:55:23.282158+0000","last_up_change":"2024-02-09T21:55:22.274974+0000","last_in_change":"2024-02-09T21:55:03.608607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-09T21:54:09.146714+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"c78f3c5b-14ab-40b7-bd39-52cb040b6532","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":4292395427},{"type":"v1","addr":"172.21.15.6:6803","nonce":4292395427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":4292395427},{"type":"v1","addr":"172.21.15.6:6805","nonce":4292395427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":4292395427},{"type":"v1","addr":"172.21.15.6:6809","nonce":4292395427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":4292395427},{"type":"v1","addr":"172.21.15.6:6807","nonce":4292395427}]},"public_addr":"172.21.15.6:6803/4292395427","cluster_addr":"172.21.15.6:6805/4292395427","heartbeat_back_addr":"172.21.15.6:6809/4292395427","heartbeat_front_addr":"172.21.15.6:6807/4292395427","state":["exists","up"]},{"osd":1,"uuid":"c2257408-632e-4e27-ae69-99046b12b405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3556298518},{"type":"v1","addr":"172.21.15.6:6811","nonce":3556298518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3556298518},{"type":"v1","addr":"172.21.15.6:6813","nonce":3556298518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":3556298518},{"type":"v1","addr":"172.21.15.6:6817","nonce":3556298518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":3556298518},{"type":"v1","addr":"172.21.15.6:6815","nonce":3556298518}]},"public_addr":"172.21.15.6:6811/3556298518","cluster_addr":"172.21.15.6:6813/3556298518","heartbeat_back_addr":"172.21.15.6:6817/3556298518","heartbeat_front_addr":"172.21.15.6:6815/3556298518","state":["exists","up"]},{"osd":2,"uuid":"e56e4fc0-614d-48e6-a2d6-758c34487540","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3201508057},{"type":"v1","addr":"172.21.15.6:6819","nonce":3201508057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3201508057},{"type":"v1","addr":"172.21.15.6:6821","nonce":3201508057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3201508057},{"type":"v1","addr":"172.21.15.6:6825","nonce":3201508057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3201508057},{"type":"v1","addr":"172.21.15.6:6823","nonce":3201508057}]},"public_addr":"172.21.15.6:6819/3201508057","cluster_addr":"172.21.15.6:6821/3201508057","heartbeat_back_addr":"172.21.15.6:6825/3201508057","heartbeat_front_addr":"172.21.15.6:6823/3201508057","state":["exists","up"]},{"osd":3,"uuid":"0b2a19ac-de1e-46f4-a3e0-5a8321672e4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":1692709589},{"type":"v1","addr":"172.21.15.90:6801","nonce":1692709589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1692709589},{"type":"v1","addr":"172.21.15.90:6803","nonce":1692709589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":1692709589},{"type":"v1","addr":"172.21.15.90:6807","nonce":1692709589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":1692709589},{"type":"v1","addr":"172.21.15.90:6805","nonce":1692709589}]},"public_addr":"172.21.15.90:6801/1692709589","cluster_addr":"172.21.15.90:6803/1692709589","heartbeat_back_addr":"172.21.15.90:6807/1692709589","heartbeat_front_addr":"172.21.15.90:6805/1692709589","state":["exists","up"]},{"osd":4,"uuid":"d79e2bb6-cebd-4e3c-8cec-5fac286838e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":204678191},{"type":"v1","addr":"172.21.15.90:6809","nonce":204678191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":204678191},{"type":"v1","addr":"172.21.15.90:6811","nonce":204678191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":204678191},{"type":"v1","addr":"172.21.15.90:6815","nonce":204678191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":204678191},{"type":"v1","addr":"172.21.15.90:6813","nonce":204678191}]},"public_addr":"172.21.15.90:6809/204678191","cluster_addr":"172.21.15.90:6811/204678191","heartbeat_back_addr":"172.21.15.90:6815/204678191","heartbeat_front_addr":"172.21.15.90:6813/204678191","state":["exists","up"]},{"osd":5,"uuid":"155a408e-ef3b-4c0a-8477-47fc6250b589","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3296441764},{"type":"v1","addr":"172.21.15.90:6817","nonce":3296441764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":3296441764},{"type":"v1","addr":"172.21.15.90:6819","nonce":3296441764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":3296441764},{"type":"v1","addr":"172.21.15.90:6823","nonce":3296441764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":3296441764},{"type":"v1","addr":"172.21.15.90:6821","nonce":3296441764}]},"public_addr":"172.21.15.90:6817/3296441764","cluster_addr":"172.21.15.90:6819/3296441764","heartbeat_back_addr":"172.21.15.90:6823/3296441764","heartbeat_front_addr":"172.21.15.90:6821/3296441764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T21:53:10.093731+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T21:53:39.528761+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T21:54:06.766402+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T21:54:30.531184+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T21:54:55.632482+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T21:55:21.026419+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:0/1297965797":"2024-02-10T21:51:56.813980+0000","172.21.15.6:0/2706037691":"2024-02-10T21:51:56.813980+0000","172.21.15.6:0/3672447393":"2024-02-10T21:51:56.813980+0000","172.21.15.6:6801/3321564517":"2024-02-10T21:51:56.813980+0000","172.21.15.6:0/1651082446":"2024-02-10T21:50:40.245871+0000","172.21.15.6:6800/1422219104":"2024-02-10T21:50:40.245871+0000","172.21.15.6:0/3369982019":"2024-02-10T21:50:07.987703+0000","172.21.15.6:6801/2325806844":"2024-02-10T21:50:07.987703+0000","172.21.15.6:6800/3321564517":"2024-02-10T21:51:56.813980+0000","172.21.15.6:0/147879205":"2024-02-10T21:50:07.987703+0000","172.21.15.6:6801/1422219104":"2024-02-10T21:50:40.245871+0000","172.21.15.6:0/2336225467":"2024-02-10T21:50:07.987703+0000","172.21.15.6:6800/2325806844":"2024-02-10T21:50:07.987703+0000","172.21.15.6:0/965195423":"2024-02-10T21:50:40.245871+0000","172.21.15.6:0/362750223":"2024-02-10T21:50:40.245871+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-09T21:55:42.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.358+0000 7f91702ee700 1 -- 172.21.15.6:0/831742638 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f91500692a0 msgr2=0x7f915006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:42.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.358+0000 7f91702ee700 1 --2- 172.21.15.6:0/831742638 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f91500692a0 0x7f915006b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f915c00b3b0 tx=0x7f915c009e30 comp rx=0 tx=0).stop 2024-02-09T21:55:42.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.358+0000 7f91702ee700 1 -- 172.21.15.6:0/831742638 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9168101e40 msgr2=0x7f9168113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:42.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.358+0000 7f91702ee700 1 --2- 172.21.15.6:0/831742638 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9168101e40 0x7f9168113a70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f915800cb10 tx=0x7f9158007f90 comp rx=0 tx=0).stop 2024-02-09T21:55:42.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.358+0000 7f91702ee700 1 -- 172.21.15.6:0/831742638 shutdown_connections 2024-02-09T21:55:42.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.358+0000 7f91702ee700 1 --2- 172.21.15.6:0/831742638 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f91500692a0 0x7f915006b750 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:42.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.358+0000 7f91702ee700 1 --2- 172.21.15.6:0/831742638 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9168101e40 0x7f9168113a70 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:42.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.358+0000 7f91702ee700 1 --2- 172.21.15.6:0/831742638 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91681027e0 0x7f9168113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:42.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.358+0000 7f91702ee700 1 -- 172.21.15.6:0/831742638 >> 172.21.15.6:0/831742638 conn(0x7f9168071d90 msgr2=0x7f9168065760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:55:42.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.358+0000 7f91702ee700 1 -- 172.21.15.6:0/831742638 shutdown_connections 2024-02-09T21:55:42.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:42.359+0000 7f91702ee700 1 -- 172.21.15.6:0/831742638 wait complete. 2024-02-09T21:55:42.977 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-09T21:55:42.978 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-09T21:55:42.979 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-09T21:55:42.979 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-09T21:55:42.979 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-09T21:55:42.980 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-09T21:55:43.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:42 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/831742638' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T21:55:43.310 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:55:43.319 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:55:43.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:42 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/831742638' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T21:55:44.067 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:43 smithi006 ceph-mon[29258]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:44.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:43 smithi090 ceph-mon[36996]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:45.040 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:55:45.041 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:55:45.041 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:55:45.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.897+0000 7f9d54dcb700 1 -- 172.21.15.6:0/2992968824 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d501038e0 msgr2=0x7f9d500fcf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:45.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.897+0000 7f9d54dcb700 1 --2- 172.21.15.6:0/2992968824 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d501038e0 0x7f9d500fcf50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9d44004940 tx=0x7f9d44009e30 comp rx=0 tx=0).stop 2024-02-09T21:55:45.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.897+0000 7f9d54dcb700 1 -- 172.21.15.6:0/2992968824 shutdown_connections 2024-02-09T21:55:45.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.897+0000 7f9d54dcb700 1 --2- 172.21.15.6:0/2992968824 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d501038e0 0x7f9d500fcf50 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:45.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.897+0000 7f9d54dcb700 1 --2- 172.21.15.6:0/2992968824 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d50102f40 0x7f9d50103310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:45.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.897+0000 7f9d54dcb700 1 -- 172.21.15.6:0/2992968824 >> 172.21.15.6:0/2992968824 conn(0x7f9d500f8af0 msgr2=0x7f9d500faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:55:45.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.897+0000 7f9d54dcb700 1 -- 172.21.15.6:0/2992968824 shutdown_connections 2024-02-09T21:55:45.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.897+0000 7f9d54dcb700 1 -- 172.21.15.6:0/2992968824 wait complete. 2024-02-09T21:55:45.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.897+0000 7f9d54dcb700 1 Processor -- start 2024-02-09T21:55:45.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.897+0000 7f9d54dcb700 1 -- start start 2024-02-09T21:55:45.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.898+0000 7f9d54dcb700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d50102f40 0x7f9d501139f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:45.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.898+0000 7f9d54dcb700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d501038e0 0x7f9d50113f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:45.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.898+0000 7f9d54dcb700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d50114580 con 0x7f9d501038e0 2024-02-09T21:55:45.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.898+0000 7f9d54dcb700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5010d8b0 con 0x7f9d50102f40 2024-02-09T21:55:45.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.898+0000 7f9d4dd9b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d501038e0 0x7f9d50113f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:45.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.898+0000 7f9d4dd9b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d501038e0 0x7f9d50113f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33142/0 (socket says 172.21.15.6:33142) 2024-02-09T21:55:45.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.898+0000 7f9d4dd9b700 1 -- 172.21.15.6:0/3196582118 learned_addr learned my addr 172.21.15.6:0/3196582118 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:55:45.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.898+0000 7f9d4e59c700 1 --2- 172.21.15.6:0/3196582118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d50102f40 0x7f9d501139f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:45.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.898+0000 7f9d4dd9b700 1 -- 172.21.15.6:0/3196582118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d50102f40 msgr2=0x7f9d501139f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:45.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.898+0000 7f9d4dd9b700 1 --2- 172.21.15.6:0/3196582118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d50102f40 0x7f9d501139f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:45.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.898+0000 7f9d4dd9b700 1 -- 172.21.15.6:0/3196582118 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d44005040 con 0x7f9d501038e0 2024-02-09T21:55:45.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.898+0000 7f9d4e59c700 1 --2- 172.21.15.6:0/3196582118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d50102f40 0x7f9d501139f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T21:55:45.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.898+0000 7f9d4dd9b700 1 --2- 172.21.15.6:0/3196582118 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d501038e0 0x7f9d50113f30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9d44009150 tx=0x7f9d44009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:45.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.899+0000 7f9d3effd700 1 -- 172.21.15.6:0/3196582118 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d4400e040 con 0x7f9d501038e0 2024-02-09T21:55:45.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.899+0000 7f9d3effd700 1 -- 172.21.15.6:0/3196582118 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9d4400bbf0 con 0x7f9d501038e0 2024-02-09T21:55:45.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.899+0000 7f9d54dcb700 1 -- 172.21.15.6:0/3196582118 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d5010db30 con 0x7f9d501038e0 2024-02-09T21:55:45.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.899+0000 7f9d3effd700 1 -- 172.21.15.6:0/3196582118 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d4401c630 con 0x7f9d501038e0 2024-02-09T21:55:45.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.899+0000 7f9d54dcb700 1 -- 172.21.15.6:0/3196582118 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d5010e070 con 0x7f9d501038e0 2024-02-09T21:55:45.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.899+0000 7f9d54dcb700 1 -- 172.21.15.6:0/3196582118 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9d5010e550 con 0x7f9d501038e0 2024-02-09T21:55:45.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.900+0000 7f9d3effd700 1 -- 172.21.15.6:0/3196582118 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f9d4400bd60 con 0x7f9d501038e0 2024-02-09T21:55:45.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.900+0000 7f9d3effd700 1 --2- 172.21.15.6:0/3196582118 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f9d380692a0 0x7f9d3806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:45.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.900+0000 7f9d3effd700 1 -- 172.21.15.6:0/3196582118 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9d440881c0 con 0x7f9d501038e0 2024-02-09T21:55:45.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.900+0000 7f9d4e59c700 1 --2- 172.21.15.6:0/3196582118 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f9d380692a0 0x7f9d3806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:45.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.900+0000 7f9d3effd700 1 --2- 172.21.15.6:0/3196582118 >> [v2:172.21.15.90:6808/204678191,v1:172.21.15.90:6809/204678191] conn(0x7f9d3806ef70 0x7f9d38071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:45.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.901+0000 7f9d3effd700 1 -- 172.21.15.6:0/3196582118 --> [v2:172.21.15.90:6808/204678191,v1:172.21.15.90:6809/204678191] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d38071a30 con 0x7f9d3806ef70 2024-02-09T21:55:45.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.901+0000 7f9d3effd700 1 -- 172.21.15.6:0/3196582118 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d4408a7b0 con 0x7f9d501038e0 2024-02-09T21:55:45.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.901+0000 7f9d4ed9d700 1 --2- 172.21.15.6:0/3196582118 >> [v2:172.21.15.90:6808/204678191,v1:172.21.15.90:6809/204678191] conn(0x7f9d3806ef70 0x7f9d38071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:45.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.901+0000 7f9d4e59c700 1 --2- 172.21.15.6:0/3196582118 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f9d380692a0 0x7f9d3806b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9d40009b80 tx=0x7f9d4000b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:45.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.901+0000 7f9d4ed9d700 1 --2- 172.21.15.6:0/3196582118 >> [v2:172.21.15.90:6808/204678191,v1:172.21.15.90:6809/204678191] conn(0x7f9d3806ef70 0x7f9d38071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:45.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.902+0000 7f9d3effd700 1 -- 172.21.15.6:0/3196582118 <== osd.4 v2:172.21.15.90:6808/204678191 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f9d38071a30 con 0x7f9d3806ef70 2024-02-09T21:55:45.915 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:55:45.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.918+0000 7f9d54dcb700 1 -- 172.21.15.6:0/3196582118 --> [v2:172.21.15.90:6808/204678191,v1:172.21.15.90:6809/204678191] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9d5005e690 con 0x7f9d3806ef70 2024-02-09T21:55:45.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.918+0000 7f9d3effd700 1 -- 172.21.15.6:0/3196582118 <== osd.4 v2:172.21.15.90:6808/204678191 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f9d5005e690 con 0x7f9d3806ef70 2024-02-09T21:55:45.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.919+0000 7f9d54dcb700 1 -- 172.21.15.6:0/3196582118 >> [v2:172.21.15.90:6808/204678191,v1:172.21.15.90:6809/204678191] conn(0x7f9d3806ef70 msgr2=0x7f9d38071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:45.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.919+0000 7f9d54dcb700 1 --2- 172.21.15.6:0/3196582118 >> [v2:172.21.15.90:6808/204678191,v1:172.21.15.90:6809/204678191] conn(0x7f9d3806ef70 0x7f9d38071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:45.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.919+0000 7f9d54dcb700 1 -- 172.21.15.6:0/3196582118 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f9d380692a0 msgr2=0x7f9d3806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:45.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.919+0000 7f9d54dcb700 1 --2- 172.21.15.6:0/3196582118 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f9d380692a0 0x7f9d3806b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9d40009b80 tx=0x7f9d4000b040 comp rx=0 tx=0).stop 2024-02-09T21:55:45.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.919+0000 7f9d54dcb700 1 -- 172.21.15.6:0/3196582118 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d501038e0 msgr2=0x7f9d50113f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:45.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.919+0000 7f9d54dcb700 1 --2- 172.21.15.6:0/3196582118 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d501038e0 0x7f9d50113f30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9d44009150 tx=0x7f9d44009e30 comp rx=0 tx=0).stop 2024-02-09T21:55:45.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.919+0000 7f9d54dcb700 1 -- 172.21.15.6:0/3196582118 shutdown_connections 2024-02-09T21:55:45.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.919+0000 7f9d54dcb700 1 --2- 172.21.15.6:0/3196582118 >> [v2:172.21.15.90:6808/204678191,v1:172.21.15.90:6809/204678191] conn(0x7f9d3806ef70 0x7f9d38071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:45.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.919+0000 7f9d54dcb700 1 --2- 172.21.15.6:0/3196582118 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f9d380692a0 0x7f9d3806b750 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:45.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.919+0000 7f9d54dcb700 1 --2- 172.21.15.6:0/3196582118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d50102f40 0x7f9d501139f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:45.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.919+0000 7f9d54dcb700 1 --2- 172.21.15.6:0/3196582118 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d501038e0 0x7f9d50113f30 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:45.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.919+0000 7f9d54dcb700 1 -- 172.21.15.6:0/3196582118 >> 172.21.15.6:0/3196582118 conn(0x7f9d500f8af0 msgr2=0x7f9d500fa4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:55:45.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.919+0000 7f9d54dcb700 1 -- 172.21.15.6:0/3196582118 shutdown_connections 2024-02-09T21:55:45.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:45.919+0000 7f9d54dcb700 1 -- 172.21.15.6:0/3196582118 wait complete. 2024-02-09T21:55:46.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:45 smithi006 ceph-mon[29258]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:46.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:45 smithi090 ceph-mon[36996]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:47.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:47 smithi006 ceph-mon[29258]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:47.704 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:47 smithi090 ceph-mon[36996]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:49.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.918+0000 7feb8c389700 1 -- 172.21.15.6:0/1186597685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb84101e40 msgr2=0x7feb84102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:49.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.918+0000 7feb8c389700 1 --2- 172.21.15.6:0/1186597685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb84101e40 0x7feb84102210 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7feb80004940 tx=0x7feb80009e30 comp rx=0 tx=0).stop 2024-02-09T21:55:49.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.919+0000 7feb8c389700 1 -- 172.21.15.6:0/1186597685 shutdown_connections 2024-02-09T21:55:49.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.919+0000 7feb8c389700 1 --2- 172.21.15.6:0/1186597685 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb841027e0 0x7feb840fbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:49.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.919+0000 7feb8c389700 1 --2- 172.21.15.6:0/1186597685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb84101e40 0x7feb84102210 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:49.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.919+0000 7feb8c389700 1 -- 172.21.15.6:0/1186597685 >> 172.21.15.6:0/1186597685 conn(0x7feb840f79d0 msgr2=0x7feb840f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:55:49.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.919+0000 7feb8c389700 1 -- 172.21.15.6:0/1186597685 shutdown_connections 2024-02-09T21:55:49.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.919+0000 7feb8c389700 1 -- 172.21.15.6:0/1186597685 wait complete. 2024-02-09T21:55:49.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.919+0000 7feb8c389700 1 Processor -- start 2024-02-09T21:55:49.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.919+0000 7feb8c389700 1 -- start start 2024-02-09T21:55:49.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.919+0000 7feb8c389700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb84101e40 0x7feb84102ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:49.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.919+0000 7feb8c389700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb841027e0 0x7feb841031e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:49.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.919+0000 7feb8c389700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb841037e0 con 0x7feb84101e40 2024-02-09T21:55:49.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.919+0000 7feb8c389700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb84103950 con 0x7feb841027e0 2024-02-09T21:55:49.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.919+0000 7feb8a125700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb84101e40 0x7feb84102ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:49.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.919+0000 7feb8a125700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb84101e40 0x7feb84102ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57854/0 (socket says 172.21.15.6:57854) 2024-02-09T21:55:49.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.919+0000 7feb8a125700 1 -- 172.21.15.6:0/211278171 learned_addr learned my addr 172.21.15.6:0/211278171 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:55:49.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.920+0000 7feb89924700 1 --2- 172.21.15.6:0/211278171 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb841027e0 0x7feb841031e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:49.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.920+0000 7feb8a125700 1 -- 172.21.15.6:0/211278171 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb841027e0 msgr2=0x7feb841031e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:49.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.920+0000 7feb8a125700 1 --2- 172.21.15.6:0/211278171 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb841027e0 0x7feb841031e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:49.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.920+0000 7feb8a125700 1 -- 172.21.15.6:0/211278171 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb80005040 con 0x7feb84101e40 2024-02-09T21:55:49.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.920+0000 7feb8a125700 1 --2- 172.21.15.6:0/211278171 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb84101e40 0x7feb84102ca0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7feb8000b3b0 tx=0x7feb80009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:49.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.920+0000 7feb7affd700 1 -- 172.21.15.6:0/211278171 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb8000e040 con 0x7feb84101e40 2024-02-09T21:55:49.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.920+0000 7feb7affd700 1 -- 172.21.15.6:0/211278171 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb8000bbf0 con 0x7feb84101e40 2024-02-09T21:55:49.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.920+0000 7feb7affd700 1 -- 172.21.15.6:0/211278171 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb8001c3f0 con 0x7feb84101e40 2024-02-09T21:55:49.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.920+0000 7feb8c389700 1 -- 172.21.15.6:0/211278171 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb84119cd0 con 0x7feb84101e40 2024-02-09T21:55:49.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.920+0000 7feb8c389700 1 -- 172.21.15.6:0/211278171 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb8411a0e0 con 0x7feb84101e40 2024-02-09T21:55:49.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.921+0000 7feb8c389700 1 -- 172.21.15.6:0/211278171 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7feb84066920 con 0x7feb84101e40 2024-02-09T21:55:49.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.922+0000 7feb7affd700 1 -- 172.21.15.6:0/211278171 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7feb8000bd60 con 0x7feb84101e40 2024-02-09T21:55:49.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.922+0000 7feb7affd700 1 --2- 172.21.15.6:0/211278171 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7feb700692a0 0x7feb7006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:49.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.922+0000 7feb7affd700 1 -- 172.21.15.6:0/211278171 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7feb800881b0 con 0x7feb84101e40 2024-02-09T21:55:49.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.922+0000 7feb89924700 1 --2- 172.21.15.6:0/211278171 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7feb700692a0 0x7feb7006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:49.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.923+0000 7feb7affd700 1 --2- 172.21.15.6:0/211278171 >> [v2:172.21.15.6:6802/4292395427,v1:172.21.15.6:6803/4292395427] conn(0x7feb7006ef70 0x7feb70071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:49.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.923+0000 7feb7affd700 1 -- 172.21.15.6:0/211278171 --> [v2:172.21.15.6:6802/4292395427,v1:172.21.15.6:6803/4292395427] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feb70071a30 con 0x7feb7006ef70 2024-02-09T21:55:49.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.923+0000 7feb7affd700 1 -- 172.21.15.6:0/211278171 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb8008a800 con 0x7feb84101e40 2024-02-09T21:55:49.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.923+0000 7feb8a926700 1 --2- 172.21.15.6:0/211278171 >> [v2:172.21.15.6:6802/4292395427,v1:172.21.15.6:6803/4292395427] conn(0x7feb7006ef70 0x7feb70071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:49.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.923+0000 7feb89924700 1 --2- 172.21.15.6:0/211278171 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7feb700692a0 0x7feb7006b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7feb841046a0 tx=0x7feb7400b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:49.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.923+0000 7feb8a926700 1 --2- 172.21.15.6:0/211278171 >> [v2:172.21.15.6:6802/4292395427,v1:172.21.15.6:6803/4292395427] conn(0x7feb7006ef70 0x7feb70071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:49.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.924+0000 7feb7affd700 1 -- 172.21.15.6:0/211278171 <== osd.0 v2:172.21.15.6:6802/4292395427 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7feb70071a30 con 0x7feb7006ef70 2024-02-09T21:55:49.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.939+0000 7feb8c389700 1 -- 172.21.15.6:0/211278171 --> [v2:172.21.15.6:6802/4292395427,v1:172.21.15.6:6803/4292395427] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7feb840fb5d0 con 0x7feb7006ef70 2024-02-09T21:55:49.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.939+0000 7feb7affd700 1 -- 172.21.15.6:0/211278171 <== osd.0 v2:172.21.15.6:6802/4292395427 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7feb840fb5d0 con 0x7feb7006ef70 2024-02-09T21:55:49.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.939+0000 7feb8c389700 1 -- 172.21.15.6:0/211278171 >> [v2:172.21.15.6:6802/4292395427,v1:172.21.15.6:6803/4292395427] conn(0x7feb7006ef70 msgr2=0x7feb70071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:49.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.939+0000 7feb8c389700 1 --2- 172.21.15.6:0/211278171 >> [v2:172.21.15.6:6802/4292395427,v1:172.21.15.6:6803/4292395427] conn(0x7feb7006ef70 0x7feb70071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:49.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.939+0000 7feb8c389700 1 -- 172.21.15.6:0/211278171 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7feb700692a0 msgr2=0x7feb7006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:49.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.939+0000 7feb8c389700 1 --2- 172.21.15.6:0/211278171 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7feb700692a0 0x7feb7006b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7feb841046a0 tx=0x7feb7400b040 comp rx=0 tx=0).stop 2024-02-09T21:55:49.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.939+0000 7feb8c389700 1 -- 172.21.15.6:0/211278171 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb84101e40 msgr2=0x7feb84102ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:49.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.939+0000 7feb8c389700 1 --2- 172.21.15.6:0/211278171 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb84101e40 0x7feb84102ca0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7feb8000b3b0 tx=0x7feb80009e30 comp rx=0 tx=0).stop 2024-02-09T21:55:49.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.939+0000 7feb8c389700 1 -- 172.21.15.6:0/211278171 shutdown_connections 2024-02-09T21:55:49.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.939+0000 7feb8c389700 1 --2- 172.21.15.6:0/211278171 >> [v2:172.21.15.6:6802/4292395427,v1:172.21.15.6:6803/4292395427] conn(0x7feb7006ef70 0x7feb70071380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:49.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.939+0000 7feb8c389700 1 --2- 172.21.15.6:0/211278171 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7feb700692a0 0x7feb7006b750 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:49.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.939+0000 7feb8c389700 1 --2- 172.21.15.6:0/211278171 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb84101e40 0x7feb84102ca0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:49.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.939+0000 7feb8c389700 1 --2- 172.21.15.6:0/211278171 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb841027e0 0x7feb841031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:49.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.939+0000 7feb8c389700 1 -- 172.21.15.6:0/211278171 >> 172.21.15.6:0/211278171 conn(0x7feb840f79d0 msgr2=0x7feb84065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:55:49.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.940+0000 7feb8c389700 1 -- 172.21.15.6:0/211278171 shutdown_connections 2024-02-09T21:55:49.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:49.940+0000 7feb8c389700 1 -- 172.21.15.6:0/211278171 wait complete. 2024-02-09T21:55:50.082 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:49 smithi006 ceph-mon[29258]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:50.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:49 smithi090 ceph-mon[36996]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:50.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.308+0000 7f6763279700 1 -- 172.21.15.6:0/2131291203 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f675c0fb570 msgr2=0x7f675c0fb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:50.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.308+0000 7f6763279700 1 --2- 172.21.15.6:0/2131291203 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f675c0fb570 0x7f675c0fb9c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6758004940 tx=0x7f6758009e30 comp rx=0 tx=0).stop 2024-02-09T21:55:50.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.308+0000 7f6763279700 1 -- 172.21.15.6:0/2131291203 shutdown_connections 2024-02-09T21:55:50.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.308+0000 7f6763279700 1 --2- 172.21.15.6:0/2131291203 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f675c0fb570 0x7f675c0fb9c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:50.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.308+0000 7f6763279700 1 --2- 172.21.15.6:0/2131291203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f675c103140 0x7f675c103510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:50.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.308+0000 7f6763279700 1 -- 172.21.15.6:0/2131291203 >> 172.21.15.6:0/2131291203 conn(0x7f675c071560 msgr2=0x7f675c071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:55:50.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.308+0000 7f6763279700 1 -- 172.21.15.6:0/2131291203 shutdown_connections 2024-02-09T21:55:50.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.308+0000 7f6763279700 1 -- 172.21.15.6:0/2131291203 wait complete. 2024-02-09T21:55:50.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.308+0000 7f6763279700 1 Processor -- start 2024-02-09T21:55:50.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.309+0000 7f6763279700 1 -- start start 2024-02-09T21:55:50.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.309+0000 7f6763279700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f675c0fb570 0x7f675c113b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:50.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.309+0000 7f6763279700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f675c103140 0x7f675c1140d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:50.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.309+0000 7f6763279700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675c114720 con 0x7f675c0fb570 2024-02-09T21:55:50.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.309+0000 7f6763279700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675c10da50 con 0x7f675c103140 2024-02-09T21:55:50.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.309+0000 7f6761015700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f675c0fb570 0x7f675c113b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:50.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.309+0000 7f6761015700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f675c0fb570 0x7f675c113b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57884/0 (socket says 172.21.15.6:57884) 2024-02-09T21:55:50.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.309+0000 7f6761015700 1 -- 172.21.15.6:0/3389162448 learned_addr learned my addr 172.21.15.6:0/3389162448 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:55:50.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.309+0000 7f6760814700 1 --2- 172.21.15.6:0/3389162448 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f675c103140 0x7f675c1140d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:50.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.309+0000 7f6761015700 1 -- 172.21.15.6:0/3389162448 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f675c103140 msgr2=0x7f675c1140d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:50.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.309+0000 7f6761015700 1 --2- 172.21.15.6:0/3389162448 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f675c103140 0x7f675c1140d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:50.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.309+0000 7f6761015700 1 -- 172.21.15.6:0/3389162448 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6758005040 con 0x7f675c0fb570 2024-02-09T21:55:50.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.309+0000 7f6761015700 1 --2- 172.21.15.6:0/3389162448 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f675c0fb570 0x7f675c113b90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f674c00c8c0 tx=0x7f674c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:50.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.309+0000 7f6751ffb700 1 -- 172.21.15.6:0/3389162448 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f674c00a760 con 0x7f675c0fb570 2024-02-09T21:55:50.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.310+0000 7f6763279700 1 -- 172.21.15.6:0/3389162448 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f675c10dd30 con 0x7f675c0fb570 2024-02-09T21:55:50.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.310+0000 7f6751ffb700 1 -- 172.21.15.6:0/3389162448 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f674c00ad90 con 0x7f675c0fb570 2024-02-09T21:55:50.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.310+0000 7f6751ffb700 1 -- 172.21.15.6:0/3389162448 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f674c0113a0 con 0x7f675c0fb570 2024-02-09T21:55:50.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.310+0000 7f6763279700 1 -- 172.21.15.6:0/3389162448 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f675c10e270 con 0x7f675c0fb570 2024-02-09T21:55:50.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.310+0000 7f6763279700 1 -- 172.21.15.6:0/3389162448 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f675c10e7c0 con 0x7f675c0fb570 2024-02-09T21:55:50.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.311+0000 7f6751ffb700 1 -- 172.21.15.6:0/3389162448 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f674c012400 con 0x7f675c0fb570 2024-02-09T21:55:50.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.311+0000 7f6751ffb700 1 --2- 172.21.15.6:0/3389162448 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f67480692b0 0x7f674806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:50.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.311+0000 7f6751ffb700 1 -- 172.21.15.6:0/3389162448 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f674c086de0 con 0x7f675c0fb570 2024-02-09T21:55:50.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.311+0000 7f6760814700 1 --2- 172.21.15.6:0/3389162448 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f67480692b0 0x7f674806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:50.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.312+0000 7f6751ffb700 1 --2- 172.21.15.6:0/3389162448 >> [v2:172.21.15.6:6810/3556298518,v1:172.21.15.6:6811/3556298518] conn(0x7f674806ef80 0x7f6748071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:50.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.312+0000 7f6751ffb700 1 -- 172.21.15.6:0/3389162448 --> [v2:172.21.15.6:6810/3556298518,v1:172.21.15.6:6811/3556298518] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6748071a40 con 0x7f674806ef80 2024-02-09T21:55:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.312+0000 7f6751ffb700 1 -- 172.21.15.6:0/3389162448 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f674c089450 con 0x7f675c0fb570 2024-02-09T21:55:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.312+0000 7f6761816700 1 --2- 172.21.15.6:0/3389162448 >> [v2:172.21.15.6:6810/3556298518,v1:172.21.15.6:6811/3556298518] conn(0x7f674806ef80 0x7f6748071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.312+0000 7f6760814700 1 --2- 172.21.15.6:0/3389162448 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f67480692b0 0x7f674806b760 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f675c10f3c0 tx=0x7f6758009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.312+0000 7f6761816700 1 --2- 172.21.15.6:0/3389162448 >> [v2:172.21.15.6:6810/3556298518,v1:172.21.15.6:6811/3556298518] conn(0x7f674806ef80 0x7f6748071390 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.313+0000 7f6751ffb700 1 -- 172.21.15.6:0/3389162448 <== osd.1 v2:172.21.15.6:6810/3556298518 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f6748071a40 con 0x7f674806ef80 2024-02-09T21:55:50.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.328+0000 7f6763279700 1 -- 172.21.15.6:0/3389162448 --> [v2:172.21.15.6:6810/3556298518,v1:172.21.15.6:6811/3556298518] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f675c073080 con 0x7f674806ef80 2024-02-09T21:55:50.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.328+0000 7f6751ffb700 1 -- 172.21.15.6:0/3389162448 <== osd.1 v2:172.21.15.6:6810/3556298518 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f675c073080 con 0x7f674806ef80 2024-02-09T21:55:50.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.329+0000 7f6763279700 1 -- 172.21.15.6:0/3389162448 >> [v2:172.21.15.6:6810/3556298518,v1:172.21.15.6:6811/3556298518] conn(0x7f674806ef80 msgr2=0x7f6748071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:50.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.329+0000 7f6763279700 1 --2- 172.21.15.6:0/3389162448 >> [v2:172.21.15.6:6810/3556298518,v1:172.21.15.6:6811/3556298518] conn(0x7f674806ef80 0x7f6748071390 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:50.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.329+0000 7f6763279700 1 -- 172.21.15.6:0/3389162448 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f67480692b0 msgr2=0x7f674806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:50.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.329+0000 7f6763279700 1 --2- 172.21.15.6:0/3389162448 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f67480692b0 0x7f674806b760 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f675c10f3c0 tx=0x7f6758009e30 comp rx=0 tx=0).stop 2024-02-09T21:55:50.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.329+0000 7f6763279700 1 -- 172.21.15.6:0/3389162448 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f675c0fb570 msgr2=0x7f675c113b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:50.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.329+0000 7f6763279700 1 --2- 172.21.15.6:0/3389162448 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f675c0fb570 0x7f675c113b90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f674c00c8c0 tx=0x7f674c0044c0 comp rx=0 tx=0).stop 2024-02-09T21:55:50.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.329+0000 7f6763279700 1 -- 172.21.15.6:0/3389162448 shutdown_connections 2024-02-09T21:55:50.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.329+0000 7f6763279700 1 --2- 172.21.15.6:0/3389162448 >> [v2:172.21.15.6:6810/3556298518,v1:172.21.15.6:6811/3556298518] conn(0x7f674806ef80 0x7f6748071390 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:50.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.329+0000 7f6763279700 1 --2- 172.21.15.6:0/3389162448 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f67480692b0 0x7f674806b760 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:50.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.329+0000 7f6763279700 1 --2- 172.21.15.6:0/3389162448 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f675c0fb570 0x7f675c113b90 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:50.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.329+0000 7f6763279700 1 --2- 172.21.15.6:0/3389162448 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f675c103140 0x7f675c1140d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:50.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.329+0000 7f6763279700 1 -- 172.21.15.6:0/3389162448 >> 172.21.15.6:0/3389162448 conn(0x7f675c071560 msgr2=0x7f675c0f97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:55:50.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.329+0000 7f6763279700 1 -- 172.21.15.6:0/3389162448 shutdown_connections 2024-02-09T21:55:50.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.329+0000 7f6763279700 1 -- 172.21.15.6:0/3389162448 wait complete. 2024-02-09T21:55:50.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.550+0000 7f21d1632700 1 -- 172.21.15.6:0/1616888673 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21cc1038f0 msgr2=0x7f21cc0fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:50.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.550+0000 7f21d1632700 1 --2- 172.21.15.6:0/1616888673 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21cc1038f0 0x7f21cc0fcf60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f21b4004940 tx=0x7f21b4009e30 comp rx=0 tx=0).stop 2024-02-09T21:55:50.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.550+0000 7f21d1632700 1 -- 172.21.15.6:0/1616888673 shutdown_connections 2024-02-09T21:55:50.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.550+0000 7f21d1632700 1 --2- 172.21.15.6:0/1616888673 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21cc1038f0 0x7f21cc0fcf60 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:50.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.550+0000 7f21d1632700 1 --2- 172.21.15.6:0/1616888673 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21cc102f50 0x7f21cc103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:50.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.550+0000 7f21d1632700 1 -- 172.21.15.6:0/1616888673 >> 172.21.15.6:0/1616888673 conn(0x7f21cc0f8ae0 msgr2=0x7f21cc0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:55:50.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.550+0000 7f21d1632700 1 -- 172.21.15.6:0/1616888673 shutdown_connections 2024-02-09T21:55:50.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.551+0000 7f21d1632700 1 -- 172.21.15.6:0/1616888673 wait complete. 2024-02-09T21:55:50.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.551+0000 7f21d1632700 1 Processor -- start 2024-02-09T21:55:50.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.551+0000 7f21d1632700 1 -- start start 2024-02-09T21:55:50.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.552+0000 7f21d1632700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21cc102f50 0x7f21cc115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:50.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.552+0000 7f21d1632700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21cc1038f0 0x7f21cc116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:50.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.552+0000 7f21d1632700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc116890 con 0x7f21cc1038f0 2024-02-09T21:55:50.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.552+0000 7f21d1632700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc10fba0 con 0x7f21cc102f50 2024-02-09T21:55:50.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.552+0000 7f21ca7fc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21cc1038f0 0x7f21cc116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:50.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.552+0000 7f21ca7fc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21cc1038f0 0x7f21cc116240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57898/0 (socket says 172.21.15.6:57898) 2024-02-09T21:55:50.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.552+0000 7f21ca7fc700 1 -- 172.21.15.6:0/1719204680 learned_addr learned my addr 172.21.15.6:0/1719204680 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:55:50.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.552+0000 7f21caffd700 1 --2- 172.21.15.6:0/1719204680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21cc102f50 0x7f21cc115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:50.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.552+0000 7f21ca7fc700 1 -- 172.21.15.6:0/1719204680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21cc102f50 msgr2=0x7f21cc115d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:50.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.552+0000 7f21ca7fc700 1 --2- 172.21.15.6:0/1719204680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21cc102f50 0x7f21cc115d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:50.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.552+0000 7f21ca7fc700 1 -- 172.21.15.6:0/1719204680 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b4005040 con 0x7f21cc1038f0 2024-02-09T21:55:50.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.552+0000 7f21ca7fc700 1 --2- 172.21.15.6:0/1719204680 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21cc1038f0 0x7f21cc116240 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f21b40090d0 tx=0x7f21b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:50.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.552+0000 7f21c37fe700 1 -- 172.21.15.6:0/1719204680 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21b400e040 con 0x7f21cc1038f0 2024-02-09T21:55:50.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.552+0000 7f21c37fe700 1 -- 172.21.15.6:0/1719204680 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f21b4009430 con 0x7f21cc1038f0 2024-02-09T21:55:50.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.552+0000 7f21d1632700 1 -- 172.21.15.6:0/1719204680 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21cc10fe20 con 0x7f21cc1038f0 2024-02-09T21:55:50.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.552+0000 7f21c37fe700 1 -- 172.21.15.6:0/1719204680 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21b401c3f0 con 0x7f21cc1038f0 2024-02-09T21:55:50.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.552+0000 7f21d1632700 1 -- 172.21.15.6:0/1719204680 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21cc110360 con 0x7f21cc1038f0 2024-02-09T21:55:50.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.553+0000 7f21d1632700 1 -- 172.21.15.6:0/1719204680 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f21cc110840 con 0x7f21cc1038f0 2024-02-09T21:55:50.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.554+0000 7f21c37fe700 1 -- 172.21.15.6:0/1719204680 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f21b400bbf0 con 0x7f21cc1038f0 2024-02-09T21:55:50.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.554+0000 7f21c37fe700 1 --2- 172.21.15.6:0/1719204680 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f21b80692a0 0x7f21b806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:50.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.554+0000 7f21c37fe700 1 -- 172.21.15.6:0/1719204680 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f21b4088110 con 0x7f21cc1038f0 2024-02-09T21:55:50.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.554+0000 7f21caffd700 1 --2- 172.21.15.6:0/1719204680 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f21b80692a0 0x7f21b806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:50.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.554+0000 7f21c37fe700 1 --2- 172.21.15.6:0/1719204680 >> [v2:172.21.15.90:6816/3296441764,v1:172.21.15.90:6817/3296441764] conn(0x7f21b806ef70 0x7f21b8071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:50.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.554+0000 7f21c37fe700 1 -- 172.21.15.6:0/1719204680 --> [v2:172.21.15.90:6816/3296441764,v1:172.21.15.90:6817/3296441764] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21b8071a30 con 0x7f21b806ef70 2024-02-09T21:55:50.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.554+0000 7f21c37fe700 1 -- 172.21.15.6:0/1719204680 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21b408a700 con 0x7f21cc1038f0 2024-02-09T21:55:50.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.555+0000 7f21caffd700 1 --2- 172.21.15.6:0/1719204680 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f21b80692a0 0x7f21b806b750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f21c4001370 tx=0x7f21c400b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:50.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.555+0000 7f21cb7fe700 1 --2- 172.21.15.6:0/1719204680 >> [v2:172.21.15.90:6816/3296441764,v1:172.21.15.90:6817/3296441764] conn(0x7f21b806ef70 0x7f21b8071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:50.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.555+0000 7f21cb7fe700 1 --2- 172.21.15.6:0/1719204680 >> [v2:172.21.15.90:6816/3296441764,v1:172.21.15.90:6817/3296441764] conn(0x7f21b806ef70 0x7f21b8071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:50.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.556+0000 7f21c37fe700 1 -- 172.21.15.6:0/1719204680 <== osd.5 v2:172.21.15.90:6816/3296441764 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f21b8071a30 con 0x7f21b806ef70 2024-02-09T21:55:50.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.571+0000 7f21d1632700 1 -- 172.21.15.6:0/1719204680 --> [v2:172.21.15.90:6816/3296441764,v1:172.21.15.90:6817/3296441764] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f21cc0fae60 con 0x7f21b806ef70 2024-02-09T21:55:50.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.572+0000 7f21c37fe700 1 -- 172.21.15.6:0/1719204680 <== osd.5 v2:172.21.15.90:6816/3296441764 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f21cc0fae60 con 0x7f21b806ef70 2024-02-09T21:55:50.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.572+0000 7f21d1632700 1 -- 172.21.15.6:0/1719204680 >> [v2:172.21.15.90:6816/3296441764,v1:172.21.15.90:6817/3296441764] conn(0x7f21b806ef70 msgr2=0x7f21b8071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:50.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.572+0000 7f21d1632700 1 --2- 172.21.15.6:0/1719204680 >> [v2:172.21.15.90:6816/3296441764,v1:172.21.15.90:6817/3296441764] conn(0x7f21b806ef70 0x7f21b8071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:50.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.572+0000 7f21d1632700 1 -- 172.21.15.6:0/1719204680 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f21b80692a0 msgr2=0x7f21b806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:50.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.572+0000 7f21d1632700 1 --2- 172.21.15.6:0/1719204680 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f21b80692a0 0x7f21b806b750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f21c4001370 tx=0x7f21c400b040 comp rx=0 tx=0).stop 2024-02-09T21:55:50.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.572+0000 7f21d1632700 1 -- 172.21.15.6:0/1719204680 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21cc1038f0 msgr2=0x7f21cc116240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:50.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.572+0000 7f21d1632700 1 --2- 172.21.15.6:0/1719204680 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21cc1038f0 0x7f21cc116240 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f21b40090d0 tx=0x7f21b4009e30 comp rx=0 tx=0).stop 2024-02-09T21:55:50.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.572+0000 7f21d1632700 1 -- 172.21.15.6:0/1719204680 shutdown_connections 2024-02-09T21:55:50.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.572+0000 7f21d1632700 1 --2- 172.21.15.6:0/1719204680 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f21b80692a0 0x7f21b806b750 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:50.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.572+0000 7f21d1632700 1 --2- 172.21.15.6:0/1719204680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21cc102f50 0x7f21cc115d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:50.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.572+0000 7f21d1632700 1 --2- 172.21.15.6:0/1719204680 >> [v2:172.21.15.90:6816/3296441764,v1:172.21.15.90:6817/3296441764] conn(0x7f21b806ef70 0x7f21b8071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:50.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.572+0000 7f21d1632700 1 --2- 172.21.15.6:0/1719204680 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21cc1038f0 0x7f21cc116240 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:50.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.572+0000 7f21d1632700 1 -- 172.21.15.6:0/1719204680 >> 172.21.15.6:0/1719204680 conn(0x7f21cc0f8ae0 msgr2=0x7f21cc0fa490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:55:50.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.572+0000 7f21d1632700 1 -- 172.21.15.6:0/1719204680 shutdown_connections 2024-02-09T21:55:50.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:50.572+0000 7f21d1632700 1 -- 172.21.15.6:0/1719204680 wait complete. 2024-02-09T21:55:51.264 INFO:teuthology.orchestra.run.smithi006.stdout:146028888077 2024-02-09T21:55:51.264 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-09T21:55:52.014 INFO:teuthology.orchestra.run.smithi006.stdout:60129542173 2024-02-09T21:55:52.014 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-09T21:55:52.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:51 smithi090 ceph-mon[36996]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:52.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:51 smithi006 ceph-mon[29258]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:52.590 INFO:teuthology.orchestra.run.smithi006.stdout:38654705699 2024-02-09T21:55:52.590 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-09T21:55:53.178 INFO:teuthology.orchestra.run.smithi006.stdout:167503724553 2024-02-09T21:55:53.178 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-09T21:55:54.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:53 smithi006 ceph-mon[29258]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:54.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:53 smithi090 ceph-mon[36996]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:54.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.894+0000 7fa6ff5af700 1 -- 172.21.15.6:0/3087771428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6f81027d0 msgr2=0x7fa6f80fbe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:54.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.894+0000 7fa6ff5af700 1 --2- 172.21.15.6:0/3087771428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6f81027d0 0x7fa6f80fbe40 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8004940 tx=0x7fa6e8009e30 comp rx=0 tx=0).stop 2024-02-09T21:55:54.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.894+0000 7fa6ff5af700 1 -- 172.21.15.6:0/3087771428 shutdown_connections 2024-02-09T21:55:54.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.894+0000 7fa6ff5af700 1 --2- 172.21.15.6:0/3087771428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6f81027d0 0x7fa6f80fbe40 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:54.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.894+0000 7fa6ff5af700 1 --2- 172.21.15.6:0/3087771428 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6f8101e30 0x7fa6f8102200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:54.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.894+0000 7fa6ff5af700 1 -- 172.21.15.6:0/3087771428 >> 172.21.15.6:0/3087771428 conn(0x7fa6f80f79c0 msgr2=0x7fa6f80f9dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:55:54.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.894+0000 7fa6ff5af700 1 -- 172.21.15.6:0/3087771428 shutdown_connections 2024-02-09T21:55:54.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.894+0000 7fa6ff5af700 1 -- 172.21.15.6:0/3087771428 wait complete. 2024-02-09T21:55:54.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.895+0000 7fa6ff5af700 1 Processor -- start 2024-02-09T21:55:54.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.895+0000 7fa6ff5af700 1 -- start start 2024-02-09T21:55:54.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.895+0000 7fa6ff5af700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6f8101e30 0x7fa6f810ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:54.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.895+0000 7fa6ff5af700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6f81027d0 0x7fa6f810f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:54.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.895+0000 7fa6ff5af700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f810f750 con 0x7fa6f8101e30 2024-02-09T21:55:54.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.895+0000 7fa6ff5af700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f810f890 con 0x7fa6f81027d0 2024-02-09T21:55:54.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.895+0000 7fa6fd34b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6f8101e30 0x7fa6f810ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:54.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.895+0000 7fa6fd34b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6f8101e30 0x7fa6f810ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57924/0 (socket says 172.21.15.6:57924) 2024-02-09T21:55:54.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.895+0000 7fa6fd34b700 1 -- 172.21.15.6:0/581592275 learned_addr learned my addr 172.21.15.6:0/581592275 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:55:54.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.896+0000 7fa6fcb4a700 1 --2- 172.21.15.6:0/581592275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6f81027d0 0x7fa6f810f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:54.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.896+0000 7fa6fd34b700 1 -- 172.21.15.6:0/581592275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6f81027d0 msgr2=0x7fa6f810f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:54.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.896+0000 7fa6fd34b700 1 --2- 172.21.15.6:0/581592275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6f81027d0 0x7fa6f810f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:54.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.896+0000 7fa6fd34b700 1 -- 172.21.15.6:0/581592275 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e8005040 con 0x7fa6f8101e30 2024-02-09T21:55:54.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.896+0000 7fa6fd34b700 1 --2- 172.21.15.6:0/581592275 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6f8101e30 0x7fa6f810ebf0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa6f400b7c0 tx=0x7fa6f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:54.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.896+0000 7fa6edffb700 1 -- 172.21.15.6:0/581592275 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6f400f6f0 con 0x7fa6f8101e30 2024-02-09T21:55:54.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.896+0000 7fa6edffb700 1 -- 172.21.15.6:0/581592275 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa6f400fd20 con 0x7fa6f8101e30 2024-02-09T21:55:54.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.896+0000 7fa6ff5af700 1 -- 172.21.15.6:0/581592275 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6f8117bc0 con 0x7fa6f8101e30 2024-02-09T21:55:54.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.896+0000 7fa6edffb700 1 -- 172.21.15.6:0/581592275 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6f40103a0 con 0x7fa6f8101e30 2024-02-09T21:55:54.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.896+0000 7fa6ff5af700 1 -- 172.21.15.6:0/581592275 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6f8118100 con 0x7fa6f8101e30 2024-02-09T21:55:54.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.897+0000 7fa6ff5af700 1 -- 172.21.15.6:0/581592275 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa6f805e610 con 0x7fa6f8101e30 2024-02-09T21:55:54.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.897+0000 7fa6edffb700 1 -- 172.21.15.6:0/581592275 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fa6f4011400 con 0x7fa6f8101e30 2024-02-09T21:55:54.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.898+0000 7fa6edffb700 1 --2- 172.21.15.6:0/581592275 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fa6e40692a0 0x7fa6e406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:54.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.898+0000 7fa6edffb700 1 -- 172.21.15.6:0/581592275 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa6f4086cb0 con 0x7fa6f8101e30 2024-02-09T21:55:54.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.898+0000 7fa6fcb4a700 1 --2- 172.21.15.6:0/581592275 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fa6e40692a0 0x7fa6e406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:54.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.898+0000 7fa6edffb700 1 --2- 172.21.15.6:0/581592275 >> [v2:172.21.15.6:6818/3201508057,v1:172.21.15.6:6819/3201508057] conn(0x7fa6e406ef70 0x7fa6e4071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:54.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.898+0000 7fa6edffb700 1 -- 172.21.15.6:0/581592275 --> [v2:172.21.15.6:6818/3201508057,v1:172.21.15.6:6819/3201508057] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa6e4071a30 con 0x7fa6e406ef70 2024-02-09T21:55:54.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.898+0000 7fa6edffb700 1 -- 172.21.15.6:0/581592275 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6f408c310 con 0x7fa6f8101e30 2024-02-09T21:55:54.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.898+0000 7fa6fdb4c700 1 --2- 172.21.15.6:0/581592275 >> [v2:172.21.15.6:6818/3201508057,v1:172.21.15.6:6819/3201508057] conn(0x7fa6e406ef70 0x7fa6e4071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:54.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.898+0000 7fa6fcb4a700 1 --2- 172.21.15.6:0/581592275 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fa6e40692a0 0x7fa6e406b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa6e800b3b0 tx=0x7fa6e8009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:54.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.899+0000 7fa6fdb4c700 1 --2- 172.21.15.6:0/581592275 >> [v2:172.21.15.6:6818/3201508057,v1:172.21.15.6:6819/3201508057] conn(0x7fa6e406ef70 0x7fa6e4071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:54.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.899+0000 7fa6edffb700 1 -- 172.21.15.6:0/581592275 <== osd.2 v2:172.21.15.6:6818/3201508057 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fa6e4071a30 con 0x7fa6e406ef70 2024-02-09T21:55:54.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.914+0000 7fa6ff5af700 1 -- 172.21.15.6:0/581592275 --> [v2:172.21.15.6:6818/3201508057,v1:172.21.15.6:6819/3201508057] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa6f805ea40 con 0x7fa6e406ef70 2024-02-09T21:55:54.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.914+0000 7fa6edffb700 1 -- 172.21.15.6:0/581592275 <== osd.2 v2:172.21.15.6:6818/3201508057 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa6f805ea40 con 0x7fa6e406ef70 2024-02-09T21:55:54.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.915+0000 7fa6ff5af700 1 -- 172.21.15.6:0/581592275 >> [v2:172.21.15.6:6818/3201508057,v1:172.21.15.6:6819/3201508057] conn(0x7fa6e406ef70 msgr2=0x7fa6e4071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:54.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.915+0000 7fa6ff5af700 1 --2- 172.21.15.6:0/581592275 >> [v2:172.21.15.6:6818/3201508057,v1:172.21.15.6:6819/3201508057] conn(0x7fa6e406ef70 0x7fa6e4071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:54.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.915+0000 7fa6ff5af700 1 -- 172.21.15.6:0/581592275 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fa6e40692a0 msgr2=0x7fa6e406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:54.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.915+0000 7fa6ff5af700 1 --2- 172.21.15.6:0/581592275 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fa6e40692a0 0x7fa6e406b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa6e800b3b0 tx=0x7fa6e8009e30 comp rx=0 tx=0).stop 2024-02-09T21:55:54.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.915+0000 7fa6ff5af700 1 -- 172.21.15.6:0/581592275 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6f8101e30 msgr2=0x7fa6f810ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:54.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.915+0000 7fa6ff5af700 1 --2- 172.21.15.6:0/581592275 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6f8101e30 0x7fa6f810ebf0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa6f400b7c0 tx=0x7fa6f4009250 comp rx=0 tx=0).stop 2024-02-09T21:55:54.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.915+0000 7fa6ff5af700 1 -- 172.21.15.6:0/581592275 shutdown_connections 2024-02-09T21:55:54.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.915+0000 7fa6ff5af700 1 --2- 172.21.15.6:0/581592275 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fa6e40692a0 0x7fa6e406b750 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:54.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.915+0000 7fa6ff5af700 1 --2- 172.21.15.6:0/581592275 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6f8101e30 0x7fa6f810ebf0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:54.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.915+0000 7fa6ff5af700 1 --2- 172.21.15.6:0/581592275 >> [v2:172.21.15.6:6818/3201508057,v1:172.21.15.6:6819/3201508057] conn(0x7fa6e406ef70 0x7fa6e4071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:54.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.915+0000 7fa6ff5af700 1 --2- 172.21.15.6:0/581592275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6f81027d0 0x7fa6f810f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:54.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.915+0000 7fa6ff5af700 1 -- 172.21.15.6:0/581592275 >> 172.21.15.6:0/581592275 conn(0x7fa6f80f79c0 msgr2=0x7fa6f80f9480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:55:54.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.915+0000 7fa6ff5af700 1 -- 172.21.15.6:0/581592275 shutdown_connections 2024-02-09T21:55:54.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:54.915+0000 7fa6ff5af700 1 -- 172.21.15.6:0/581592275 wait complete. 2024-02-09T21:55:55.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.111+0000 7f1884337700 1 -- 172.21.15.6:0/1206288724 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f187c102800 msgr2=0x7f187c0fbe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:55.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.111+0000 7f1884337700 1 --2- 172.21.15.6:0/1206288724 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f187c102800 0x7f187c0fbe70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1878004940 tx=0x7f1878009e30 comp rx=0 tx=0).stop 2024-02-09T21:55:55.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.111+0000 7f1884337700 1 -- 172.21.15.6:0/1206288724 shutdown_connections 2024-02-09T21:55:55.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.111+0000 7f1884337700 1 --2- 172.21.15.6:0/1206288724 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f187c102800 0x7f187c0fbe70 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:55.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.111+0000 7f1884337700 1 --2- 172.21.15.6:0/1206288724 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f187c101e60 0x7f187c102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:55.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.111+0000 7f1884337700 1 -- 172.21.15.6:0/1206288724 >> 172.21.15.6:0/1206288724 conn(0x7f187c0f7a10 msgr2=0x7f187c0f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:55:55.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.111+0000 7f1884337700 1 -- 172.21.15.6:0/1206288724 shutdown_connections 2024-02-09T21:55:55.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.111+0000 7f1884337700 1 -- 172.21.15.6:0/1206288724 wait complete. 2024-02-09T21:55:55.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.112+0000 7f1884337700 1 Processor -- start 2024-02-09T21:55:55.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.112+0000 7f1884337700 1 -- start start 2024-02-09T21:55:55.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.112+0000 7f1884337700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f187c101e60 0x7f187c10eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:55.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.112+0000 7f1884337700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f187c102800 0x7f187c10f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:55.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.112+0000 7f1884337700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187c10f650 con 0x7f187c102800 2024-02-09T21:55:55.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.112+0000 7f1884337700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187c10f7c0 con 0x7f187c101e60 2024-02-09T21:55:55.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.112+0000 7f18818d2700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f187c102800 0x7f187c10f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:55.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.112+0000 7f18818d2700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f187c102800 0x7f187c10f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57946/0 (socket says 172.21.15.6:57946) 2024-02-09T21:55:55.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.112+0000 7f18818d2700 1 -- 172.21.15.6:0/2438223562 learned_addr learned my addr 172.21.15.6:0/2438223562 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:55:55.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.112+0000 7f18820d3700 1 --2- 172.21.15.6:0/2438223562 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f187c101e60 0x7f187c10eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:55.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.112+0000 7f18818d2700 1 -- 172.21.15.6:0/2438223562 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f187c101e60 msgr2=0x7f187c10eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:55.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.112+0000 7f18818d2700 1 --2- 172.21.15.6:0/2438223562 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f187c101e60 0x7f187c10eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:55.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.112+0000 7f18818d2700 1 -- 172.21.15.6:0/2438223562 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1878005040 con 0x7f187c102800 2024-02-09T21:55:55.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.112+0000 7f18818d2700 1 --2- 172.21.15.6:0/2438223562 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f187c102800 0x7f187c10f0c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f187800b3b0 tx=0x7f1878009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:55.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.113+0000 7f1872ffd700 1 -- 172.21.15.6:0/2438223562 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f187800e040 con 0x7f187c102800 2024-02-09T21:55:55.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.113+0000 7f1872ffd700 1 -- 172.21.15.6:0/2438223562 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f187800bbf0 con 0x7f187c102800 2024-02-09T21:55:55.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.113+0000 7f1884337700 1 -- 172.21.15.6:0/2438223562 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f187c117af0 con 0x7f187c102800 2024-02-09T21:55:55.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.113+0000 7f1872ffd700 1 -- 172.21.15.6:0/2438223562 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f187801c430 con 0x7f187c102800 2024-02-09T21:55:55.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.113+0000 7f1884337700 1 -- 172.21.15.6:0/2438223562 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f187c117fb0 con 0x7f187c102800 2024-02-09T21:55:55.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.113+0000 7f1884337700 1 -- 172.21.15.6:0/2438223562 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f187c05e690 con 0x7f187c102800 2024-02-09T21:55:55.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.114+0000 7f1872ffd700 1 -- 172.21.15.6:0/2438223562 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f187800bd60 con 0x7f187c102800 2024-02-09T21:55:55.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.114+0000 7f1872ffd700 1 --2- 172.21.15.6:0/2438223562 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f18680692a0 0x7f186806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:55.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.115+0000 7f1872ffd700 1 -- 172.21.15.6:0/2438223562 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1878088cb0 con 0x7f187c102800 2024-02-09T21:55:55.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.115+0000 7f18820d3700 1 --2- 172.21.15.6:0/2438223562 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f18680692a0 0x7f186806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:55.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.115+0000 7f1872ffd700 1 --2- 172.21.15.6:0/2438223562 >> [v2:172.21.15.90:6800/1692709589,v1:172.21.15.90:6801/1692709589] conn(0x7f186806ef70 0x7f1868071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:55:55.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.115+0000 7f1872ffd700 1 -- 172.21.15.6:0/2438223562 --> [v2:172.21.15.90:6800/1692709589,v1:172.21.15.90:6801/1692709589] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1868071a30 con 0x7f186806ef70 2024-02-09T21:55:55.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.115+0000 7f1872ffd700 1 -- 172.21.15.6:0/2438223562 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18780890b0 con 0x7f187c102800 2024-02-09T21:55:55.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.115+0000 7f18828d4700 1 --2- 172.21.15.6:0/2438223562 >> [v2:172.21.15.90:6800/1692709589,v1:172.21.15.90:6801/1692709589] conn(0x7f186806ef70 0x7f1868071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:55:55.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.115+0000 7f18820d3700 1 --2- 172.21.15.6:0/2438223562 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f18680692a0 0x7f186806b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f186c007fd0 tx=0x7f186c009040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:55.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.115+0000 7f18828d4700 1 --2- 172.21.15.6:0/2438223562 >> [v2:172.21.15.90:6800/1692709589,v1:172.21.15.90:6801/1692709589] conn(0x7f186806ef70 0x7f1868071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:55:55.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.116+0000 7f1872ffd700 1 -- 172.21.15.6:0/2438223562 <== osd.3 v2:172.21.15.90:6800/1692709589 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f1868071a30 con 0x7f186806ef70 2024-02-09T21:55:55.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.131+0000 7f1884337700 1 -- 172.21.15.6:0/2438223562 --> [v2:172.21.15.90:6800/1692709589,v1:172.21.15.90:6801/1692709589] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f187c05eac0 con 0x7f186806ef70 2024-02-09T21:55:55.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.132+0000 7f1872ffd700 1 -- 172.21.15.6:0/2438223562 <== osd.3 v2:172.21.15.90:6800/1692709589 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f187c05eac0 con 0x7f186806ef70 2024-02-09T21:55:55.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.132+0000 7f1884337700 1 -- 172.21.15.6:0/2438223562 >> [v2:172.21.15.90:6800/1692709589,v1:172.21.15.90:6801/1692709589] conn(0x7f186806ef70 msgr2=0x7f1868071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:55.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.132+0000 7f1884337700 1 --2- 172.21.15.6:0/2438223562 >> [v2:172.21.15.90:6800/1692709589,v1:172.21.15.90:6801/1692709589] conn(0x7f186806ef70 0x7f1868071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:55.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.132+0000 7f1884337700 1 -- 172.21.15.6:0/2438223562 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f18680692a0 msgr2=0x7f186806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:55.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.132+0000 7f1884337700 1 --2- 172.21.15.6:0/2438223562 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f18680692a0 0x7f186806b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f186c007fd0 tx=0x7f186c009040 comp rx=0 tx=0).stop 2024-02-09T21:55:55.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.132+0000 7f1884337700 1 -- 172.21.15.6:0/2438223562 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f187c102800 msgr2=0x7f187c10f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:55:55.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.132+0000 7f1884337700 1 --2- 172.21.15.6:0/2438223562 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f187c102800 0x7f187c10f0c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f187800b3b0 tx=0x7f1878009e30 comp rx=0 tx=0).stop 2024-02-09T21:55:55.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.132+0000 7f1884337700 1 -- 172.21.15.6:0/2438223562 shutdown_connections 2024-02-09T21:55:55.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.132+0000 7f1884337700 1 --2- 172.21.15.6:0/2438223562 >> [v2:172.21.15.90:6800/1692709589,v1:172.21.15.90:6801/1692709589] conn(0x7f186806ef70 0x7f1868071380 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:55.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.132+0000 7f1884337700 1 --2- 172.21.15.6:0/2438223562 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f18680692a0 0x7f186806b750 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:55.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.132+0000 7f1884337700 1 --2- 172.21.15.6:0/2438223562 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f187c101e60 0x7f187c10eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:55.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.132+0000 7f1884337700 1 --2- 172.21.15.6:0/2438223562 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f187c102800 0x7f187c10f0c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:55:55.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.132+0000 7f1884337700 1 -- 172.21.15.6:0/2438223562 >> 172.21.15.6:0/2438223562 conn(0x7f187c0f7a10 msgr2=0x7f187c0f9530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:55:55.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.132+0000 7f1884337700 1 -- 172.21.15.6:0/2438223562 shutdown_connections 2024-02-09T21:55:55.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:55:55.132+0000 7f1884337700 1 -- 172.21.15.6:0/2438223562 wait complete. 2024-02-09T21:55:55.345 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:55:55.803 INFO:teuthology.orchestra.run.smithi006.stdout:111669149715 2024-02-09T21:55:55.803 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-09T21:55:56.034 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:55 smithi006 ceph-mon[29258]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:56.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:55 smithi090 ceph-mon[36996]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:56.354 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:55:56.354 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:55:56.734 INFO:teuthology.orchestra.run.smithi006.stdout:81604378649 2024-02-09T21:55:56.734 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-09T21:55:57.589 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:55:57.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:57 smithi006 ceph-mon[29258]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:57.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:57 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:55:57.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:57 smithi090 ceph-mon[36996]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:55:57.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:57 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:55:59.506 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:56:00.106 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:55:59 smithi006 ceph-mon[29258]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:00.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:55:59 smithi090 ceph-mon[36996]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:01.668 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:56:01.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.670+0000 7f226cae4700 1 -- 172.21.15.6:0/3594167137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2268102f50 msgr2=0x7f2268103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:01.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.670+0000 7f226cae4700 1 --2- 172.21.15.6:0/3594167137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2268102f50 0x7f2268103320 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2258004940 tx=0x7f2258009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:01.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.670+0000 7f226cae4700 1 -- 172.21.15.6:0/3594167137 shutdown_connections 2024-02-09T21:56:01.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.670+0000 7f226cae4700 1 --2- 172.21.15.6:0/3594167137 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22681038f0 0x7f22680fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:01.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.670+0000 7f226cae4700 1 --2- 172.21.15.6:0/3594167137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2268102f50 0x7f2268103320 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:01.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.670+0000 7f226cae4700 1 -- 172.21.15.6:0/3594167137 >> 172.21.15.6:0/3594167137 conn(0x7f22680f8ae0 msgr2=0x7f22680faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:01.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.670+0000 7f226cae4700 1 -- 172.21.15.6:0/3594167137 shutdown_connections 2024-02-09T21:56:01.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.670+0000 7f226cae4700 1 -- 172.21.15.6:0/3594167137 wait complete. 2024-02-09T21:56:01.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.671+0000 7f226cae4700 1 Processor -- start 2024-02-09T21:56:01.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.671+0000 7f226cae4700 1 -- start start 2024-02-09T21:56:01.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.671+0000 7f226cae4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2268102f50 0x7f2268070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:01.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.671+0000 7f226cae4700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22681038f0 0x7f22680707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:01.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.671+0000 7f226cae4700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2268070dd0 con 0x7f22681038f0 2024-02-09T21:56:01.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.671+0000 7f226cae4700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2268070f40 con 0x7f2268102f50 2024-02-09T21:56:01.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.671+0000 7f2265d9b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22681038f0 0x7f22680707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:01.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.671+0000 7f2265d9b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22681038f0 0x7f22680707d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56756/0 (socket says 172.21.15.6:56756) 2024-02-09T21:56:01.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.672+0000 7f2265d9b700 1 -- 172.21.15.6:0/615142128 learned_addr learned my addr 172.21.15.6:0/615142128 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:56:01.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.672+0000 7f226659c700 1 --2- 172.21.15.6:0/615142128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2268102f50 0x7f2268070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:01.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.672+0000 7f2265d9b700 1 -- 172.21.15.6:0/615142128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2268102f50 msgr2=0x7f2268070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:01.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.672+0000 7f2265d9b700 1 --2- 172.21.15.6:0/615142128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2268102f50 0x7f2268070290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:01.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.672+0000 7f2265d9b700 1 -- 172.21.15.6:0/615142128 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2258005040 con 0x7f22681038f0 2024-02-09T21:56:01.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.672+0000 7f2265d9b700 1 --2- 172.21.15.6:0/615142128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22681038f0 0x7f22680707d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f225c00eb40 tx=0x7f225c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:01.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.672+0000 7f2256ffd700 1 -- 172.21.15.6:0/615142128 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f225c007cc0 con 0x7f22681038f0 2024-02-09T21:56:01.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.672+0000 7f226cae4700 1 -- 172.21.15.6:0/615142128 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2268119c30 con 0x7f22681038f0 2024-02-09T21:56:01.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.672+0000 7f2256ffd700 1 -- 172.21.15.6:0/615142128 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f225c00f040 con 0x7f22681038f0 2024-02-09T21:56:01.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.672+0000 7f2256ffd700 1 -- 172.21.15.6:0/615142128 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f225c00c9d0 con 0x7f22681038f0 2024-02-09T21:56:01.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.673+0000 7f226cae4700 1 -- 172.21.15.6:0/615142128 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f226811a090 con 0x7f22681038f0 2024-02-09T21:56:01.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.674+0000 7f226cae4700 1 -- 172.21.15.6:0/615142128 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22680fc6e0 con 0x7f22681038f0 2024-02-09T21:56:01.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.674+0000 7f2256ffd700 1 -- 172.21.15.6:0/615142128 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f225c00cc40 con 0x7f22681038f0 2024-02-09T21:56:01.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.674+0000 7f2256ffd700 1 --2- 172.21.15.6:0/615142128 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f22500692b0 0x7f225006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:01.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.675+0000 7f2256ffd700 1 -- 172.21.15.6:0/615142128 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f225c086890 con 0x7f22681038f0 2024-02-09T21:56:01.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.675+0000 7f226659c700 1 --2- 172.21.15.6:0/615142128 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f22500692b0 0x7f225006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:01.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.675+0000 7f226659c700 1 --2- 172.21.15.6:0/615142128 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f22500692b0 0x7f225006b760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2258009fd0 tx=0x7f2258009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:01.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.679+0000 7f2256ffd700 1 -- 172.21.15.6:0/615142128 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f225c04dbd0 con 0x7f22681038f0 2024-02-09T21:56:01.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.883+0000 7f226cae4700 1 -- 172.21.15.6:0/615142128 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f226810a050 con 0x7f22681038f0 2024-02-09T21:56:01.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.884+0000 7f2256ffd700 1 -- 172.21.15.6:0/615142128 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f225c019090 con 0x7f22681038f0 2024-02-09T21:56:01.888 INFO:teuthology.orchestra.run.smithi006.stdout:60129542175 2024-02-09T21:56:01.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.886+0000 7f226cae4700 1 -- 172.21.15.6:0/615142128 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f22500692b0 msgr2=0x7f225006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:01.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.886+0000 7f226cae4700 1 --2- 172.21.15.6:0/615142128 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f22500692b0 0x7f225006b760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2258009fd0 tx=0x7f2258009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:01.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.886+0000 7f226cae4700 1 -- 172.21.15.6:0/615142128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22681038f0 msgr2=0x7f22680707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:01.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.886+0000 7f226cae4700 1 --2- 172.21.15.6:0/615142128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22681038f0 0x7f22680707d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f225c00eb40 tx=0x7f225c00b040 comp rx=0 tx=0).stop 2024-02-09T21:56:01.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.886+0000 7f226cae4700 1 -- 172.21.15.6:0/615142128 shutdown_connections 2024-02-09T21:56:01.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.886+0000 7f226cae4700 1 --2- 172.21.15.6:0/615142128 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f22500692b0 0x7f225006b760 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:01.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.886+0000 7f226cae4700 1 --2- 172.21.15.6:0/615142128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2268102f50 0x7f2268070290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:01.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.886+0000 7f226cae4700 1 --2- 172.21.15.6:0/615142128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22681038f0 0x7f22680707d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:01.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.886+0000 7f226cae4700 1 -- 172.21.15.6:0/615142128 >> 172.21.15.6:0/615142128 conn(0x7f22680f8ae0 msgr2=0x7f22680fa400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:01.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.886+0000 7f226cae4700 1 -- 172.21.15.6:0/615142128 shutdown_connections 2024-02-09T21:56:01.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:01.886+0000 7f226cae4700 1 -- 172.21.15.6:0/615142128 wait complete. 2024-02-09T21:56:01.971 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:01 smithi090 ceph-mon[36996]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:01.971 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:01 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/615142128' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-09T21:56:02.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:01 smithi006 ceph-mon[29258]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:02.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:01 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/615142128' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-09T21:56:04.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:03 smithi090 ceph-mon[36996]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:04.258 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:03 smithi006 ceph-mon[29258]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:04.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.356+0000 7fd2c1430700 1 -- 172.21.15.6:0/3443146864 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2bc101e60 msgr2=0x7fd2bc102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:04.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.356+0000 7fd2c1430700 1 --2- 172.21.15.6:0/3443146864 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2bc101e60 0x7fd2bc102230 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac004940 tx=0x7fd2ac009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:04.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.356+0000 7fd2c1430700 1 -- 172.21.15.6:0/3443146864 shutdown_connections 2024-02-09T21:56:04.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.356+0000 7fd2c1430700 1 --2- 172.21.15.6:0/3443146864 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2bc102800 0x7fd2bc0fbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:04.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.356+0000 7fd2c1430700 1 --2- 172.21.15.6:0/3443146864 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2bc101e60 0x7fd2bc102230 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:04.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.356+0000 7fd2c1430700 1 -- 172.21.15.6:0/3443146864 >> 172.21.15.6:0/3443146864 conn(0x7fd2bc0f7a10 msgr2=0x7fd2bc0f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:04.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.356+0000 7fd2c1430700 1 -- 172.21.15.6:0/3443146864 shutdown_connections 2024-02-09T21:56:04.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.356+0000 7fd2c1430700 1 -- 172.21.15.6:0/3443146864 wait complete. 2024-02-09T21:56:04.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.357+0000 7fd2c1430700 1 Processor -- start 2024-02-09T21:56:04.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.357+0000 7fd2c1430700 1 -- start start 2024-02-09T21:56:04.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.357+0000 7fd2c1430700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2bc101e60 0x7fd2bc112a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:04.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.357+0000 7fd2c1430700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2bc102800 0x7fd2bc112f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:04.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.357+0000 7fd2c1430700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc1135a0 con 0x7fd2bc102800 2024-02-09T21:56:04.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.357+0000 7fd2c1430700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc10c8d0 con 0x7fd2bc101e60 2024-02-09T21:56:04.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.358+0000 7fd2ba7fc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2bc102800 0x7fd2bc112f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:04.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.358+0000 7fd2ba7fc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2bc102800 0x7fd2bc112f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56776/0 (socket says 172.21.15.6:56776) 2024-02-09T21:56:04.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.358+0000 7fd2ba7fc700 1 -- 172.21.15.6:0/1021162900 learned_addr learned my addr 172.21.15.6:0/1021162900 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:56:04.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.358+0000 7fd2baffd700 1 --2- 172.21.15.6:0/1021162900 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2bc101e60 0x7fd2bc112a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:04.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.358+0000 7fd2ba7fc700 1 -- 172.21.15.6:0/1021162900 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2bc101e60 msgr2=0x7fd2bc112a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:04.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.358+0000 7fd2ba7fc700 1 --2- 172.21.15.6:0/1021162900 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2bc101e60 0x7fd2bc112a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:04.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.358+0000 7fd2ba7fc700 1 -- 172.21.15.6:0/1021162900 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ac005040 con 0x7fd2bc102800 2024-02-09T21:56:04.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.358+0000 7fd2ba7fc700 1 --2- 172.21.15.6:0/1021162900 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2bc102800 0x7fd2bc112f50 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4002a50 tx=0x7fd2a4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:04.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.358+0000 7fd2b37fe700 1 -- 172.21.15.6:0/1021162900 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2a4007cc0 con 0x7fd2bc102800 2024-02-09T21:56:04.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.358+0000 7fd2c1430700 1 -- 172.21.15.6:0/1021162900 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2bc10cbb0 con 0x7fd2bc102800 2024-02-09T21:56:04.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.358+0000 7fd2b37fe700 1 -- 172.21.15.6:0/1021162900 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd2a400f040 con 0x7fd2bc102800 2024-02-09T21:56:04.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.358+0000 7fd2b37fe700 1 -- 172.21.15.6:0/1021162900 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2a400d9c0 con 0x7fd2bc102800 2024-02-09T21:56:04.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.358+0000 7fd2c1430700 1 -- 172.21.15.6:0/1021162900 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2bc10d0f0 con 0x7fd2bc102800 2024-02-09T21:56:04.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.359+0000 7fd2c1430700 1 -- 172.21.15.6:0/1021162900 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2bc0ff170 con 0x7fd2bc102800 2024-02-09T21:56:04.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.359+0000 7fd2b37fe700 1 -- 172.21.15.6:0/1021162900 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fd2a4009730 con 0x7fd2bc102800 2024-02-09T21:56:04.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.360+0000 7fd2b37fe700 1 --2- 172.21.15.6:0/1021162900 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fd2a80692b0 0x7fd2a806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:04.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.360+0000 7fd2baffd700 1 --2- 172.21.15.6:0/1021162900 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fd2a80692b0 0x7fd2a806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:04.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.360+0000 7fd2b37fe700 1 -- 172.21.15.6:0/1021162900 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd2a4086bf0 con 0x7fd2bc102800 2024-02-09T21:56:04.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.360+0000 7fd2baffd700 1 --2- 172.21.15.6:0/1021162900 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fd2a80692b0 0x7fd2a806b760 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac00b3b0 tx=0x7fd2ac009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:04.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.364+0000 7fd2b37fe700 1 -- 172.21.15.6:0/1021162900 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd2a4056c70 con 0x7fd2bc102800 2024-02-09T21:56:04.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.559+0000 7fd2c1430700 1 -- 172.21.15.6:0/1021162900 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fd2bc108f60 con 0x7fd2bc102800 2024-02-09T21:56:04.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.560+0000 7fd2b37fe700 1 -- 172.21.15.6:0/1021162900 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd2a4056800 con 0x7fd2bc102800 2024-02-09T21:56:04.563 INFO:teuthology.orchestra.run.smithi006.stdout:38654705701 2024-02-09T21:56:04.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.561+0000 7fd2c1430700 1 -- 172.21.15.6:0/1021162900 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fd2a80692b0 msgr2=0x7fd2a806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:04.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.561+0000 7fd2c1430700 1 --2- 172.21.15.6:0/1021162900 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fd2a80692b0 0x7fd2a806b760 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac00b3b0 tx=0x7fd2ac009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:04.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.561+0000 7fd2c1430700 1 -- 172.21.15.6:0/1021162900 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2bc102800 msgr2=0x7fd2bc112f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:04.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.561+0000 7fd2c1430700 1 --2- 172.21.15.6:0/1021162900 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2bc102800 0x7fd2bc112f50 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4002a50 tx=0x7fd2a4004600 comp rx=0 tx=0).stop 2024-02-09T21:56:04.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.561+0000 7fd2c1430700 1 -- 172.21.15.6:0/1021162900 shutdown_connections 2024-02-09T21:56:04.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.561+0000 7fd2c1430700 1 --2- 172.21.15.6:0/1021162900 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fd2a80692b0 0x7fd2a806b760 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:04.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.561+0000 7fd2c1430700 1 --2- 172.21.15.6:0/1021162900 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2bc101e60 0x7fd2bc112a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:04.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.561+0000 7fd2c1430700 1 --2- 172.21.15.6:0/1021162900 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2bc102800 0x7fd2bc112f50 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:04.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.561+0000 7fd2c1430700 1 -- 172.21.15.6:0/1021162900 >> 172.21.15.6:0/1021162900 conn(0x7fd2bc0f7a10 msgr2=0x7fd2bc0f9560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:04.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.562+0000 7fd2c1430700 1 -- 172.21.15.6:0/1021162900 shutdown_connections 2024-02-09T21:56:04.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:04.562+0000 7fd2c1430700 1 -- 172.21.15.6:0/1021162900 wait complete. 2024-02-09T21:56:05.197 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:04 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/1021162900' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-09T21:56:05.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:04 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/1021162900' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-09T21:56:06.119 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:05 smithi006 ceph-mon[29258]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:06.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:05 smithi090 ceph-mon[36996]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:06.569 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705699 got 38654705701 for osd.0 2024-02-09T21:56:06.570 DEBUG:teuthology.parallel:result is None 2024-02-09T21:56:06.694 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542173 got 60129542175 for osd.1 2024-02-09T21:56:06.695 DEBUG:teuthology.parallel:result is None 2024-02-09T21:56:06.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.877+0000 7eff1869e700 1 -- 172.21.15.6:0/153093479 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff10101e40 msgr2=0x7eff10102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:06.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.877+0000 7eff1869e700 1 --2- 172.21.15.6:0/153093479 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff10101e40 0x7eff10102210 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7eff04004940 tx=0x7eff04009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:06.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.877+0000 7eff1869e700 1 -- 172.21.15.6:0/153093479 shutdown_connections 2024-02-09T21:56:06.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.877+0000 7eff1869e700 1 --2- 172.21.15.6:0/153093479 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff101027e0 0x7eff100fbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:06.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.877+0000 7eff1869e700 1 --2- 172.21.15.6:0/153093479 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff10101e40 0x7eff10102210 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:06.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.877+0000 7eff1869e700 1 -- 172.21.15.6:0/153093479 >> 172.21.15.6:0/153093479 conn(0x7eff10071d90 msgr2=0x7eff10072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:06.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.877+0000 7eff1869e700 1 -- 172.21.15.6:0/153093479 shutdown_connections 2024-02-09T21:56:06.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.877+0000 7eff1869e700 1 -- 172.21.15.6:0/153093479 wait complete. 2024-02-09T21:56:06.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.878+0000 7eff1869e700 1 Processor -- start 2024-02-09T21:56:06.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.878+0000 7eff1869e700 1 -- start start 2024-02-09T21:56:06.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.878+0000 7eff1869e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff10101e40 0x7eff1010eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:06.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.878+0000 7eff1869e700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff101027e0 0x7eff1010f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:06.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.878+0000 7eff1869e700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1010f650 con 0x7eff101027e0 2024-02-09T21:56:06.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.878+0000 7eff1869e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1010f7c0 con 0x7eff10101e40 2024-02-09T21:56:06.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.878+0000 7eff15c39700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff101027e0 0x7eff1010f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:06.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.878+0000 7eff15c39700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff101027e0 0x7eff1010f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56820/0 (socket says 172.21.15.6:56820) 2024-02-09T21:56:06.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.878+0000 7eff15c39700 1 -- 172.21.15.6:0/2082415614 learned_addr learned my addr 172.21.15.6:0/2082415614 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:56:06.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.878+0000 7eff1643a700 1 --2- 172.21.15.6:0/2082415614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff10101e40 0x7eff1010eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:06.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.878+0000 7eff15c39700 1 -- 172.21.15.6:0/2082415614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff10101e40 msgr2=0x7eff1010eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:06.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.878+0000 7eff15c39700 1 --2- 172.21.15.6:0/2082415614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff10101e40 0x7eff1010eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:06.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.878+0000 7eff15c39700 1 -- 172.21.15.6:0/2082415614 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff04005040 con 0x7eff101027e0 2024-02-09T21:56:06.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.878+0000 7eff15c39700 1 --2- 172.21.15.6:0/2082415614 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff101027e0 0x7eff1010f030 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7eff0c00b7c0 tx=0x7eff0c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:06.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.878+0000 7eff02ffd700 1 -- 172.21.15.6:0/2082415614 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff0c00f6f0 con 0x7eff101027e0 2024-02-09T21:56:06.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.879+0000 7eff02ffd700 1 -- 172.21.15.6:0/2082415614 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7eff0c00fd20 con 0x7eff101027e0 2024-02-09T21:56:06.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.879+0000 7eff1869e700 1 -- 172.21.15.6:0/2082415614 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff10117b10 con 0x7eff101027e0 2024-02-09T21:56:06.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.879+0000 7eff02ffd700 1 -- 172.21.15.6:0/2082415614 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff0c0103a0 con 0x7eff101027e0 2024-02-09T21:56:06.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.879+0000 7eff1869e700 1 -- 172.21.15.6:0/2082415614 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff10118020 con 0x7eff101027e0 2024-02-09T21:56:06.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.880+0000 7eff1869e700 1 -- 172.21.15.6:0/2082415614 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff100738b0 con 0x7eff101027e0 2024-02-09T21:56:06.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.880+0000 7eff02ffd700 1 -- 172.21.15.6:0/2082415614 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7eff0c011400 con 0x7eff101027e0 2024-02-09T21:56:06.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.880+0000 7eff02ffd700 1 --2- 172.21.15.6:0/2082415614 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7efefc0692a0 0x7efefc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:06.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.881+0000 7eff02ffd700 1 -- 172.21.15.6:0/2082415614 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7eff0c086ba0 con 0x7eff101027e0 2024-02-09T21:56:06.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.881+0000 7eff1643a700 1 --2- 172.21.15.6:0/2082415614 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7efefc0692a0 0x7efefc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:06.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.881+0000 7eff1643a700 1 --2- 172.21.15.6:0/2082415614 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7efefc0692a0 0x7efefc06b750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7eff04004560 tx=0x7eff04009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:06.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.885+0000 7eff02ffd700 1 -- 172.21.15.6:0/2082415614 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7eff0c0532b0 con 0x7eff101027e0 2024-02-09T21:56:07.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.079+0000 7eff1869e700 1 -- 172.21.15.6:0/2082415614 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7eff10108fb0 con 0x7eff101027e0 2024-02-09T21:56:07.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.080+0000 7eff02ffd700 1 -- 172.21.15.6:0/2082415614 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7eff0c0568d0 con 0x7eff101027e0 2024-02-09T21:56:07.084 INFO:teuthology.orchestra.run.smithi006.stdout:111669149717 2024-02-09T21:56:07.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.082+0000 7eff1869e700 1 -- 172.21.15.6:0/2082415614 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7efefc0692a0 msgr2=0x7efefc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:07.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.082+0000 7eff1869e700 1 --2- 172.21.15.6:0/2082415614 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7efefc0692a0 0x7efefc06b750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7eff04004560 tx=0x7eff04009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:07.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.082+0000 7eff1869e700 1 -- 172.21.15.6:0/2082415614 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff101027e0 msgr2=0x7eff1010f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:07.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.082+0000 7eff1869e700 1 --2- 172.21.15.6:0/2082415614 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff101027e0 0x7eff1010f030 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7eff0c00b7c0 tx=0x7eff0c009220 comp rx=0 tx=0).stop 2024-02-09T21:56:07.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.082+0000 7eff1869e700 1 -- 172.21.15.6:0/2082415614 shutdown_connections 2024-02-09T21:56:07.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.082+0000 7eff1869e700 1 --2- 172.21.15.6:0/2082415614 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7efefc0692a0 0x7efefc06b750 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:07.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.082+0000 7eff1869e700 1 --2- 172.21.15.6:0/2082415614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff10101e40 0x7eff1010eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:07.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.082+0000 7eff1869e700 1 --2- 172.21.15.6:0/2082415614 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff101027e0 0x7eff1010f030 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:07.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.082+0000 7eff1869e700 1 -- 172.21.15.6:0/2082415614 >> 172.21.15.6:0/2082415614 conn(0x7eff10071d90 msgr2=0x7eff100fad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:07.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.082+0000 7eff1869e700 1 -- 172.21.15.6:0/2082415614 shutdown_connections 2024-02-09T21:56:07.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.082+0000 7eff1869e700 1 -- 172.21.15.6:0/2082415614 wait complete. 2024-02-09T21:56:07.132 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:06 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/2951739961' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-09T21:56:07.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.139+0000 7f700d47b700 1 -- 172.21.15.6:0/3685867731 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7008101e60 msgr2=0x7f7008102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:07.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.139+0000 7f700d47b700 1 --2- 172.21.15.6:0/3685867731 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7008101e60 0x7f7008102230 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8004940 tx=0x7f6ff8009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:07.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.139+0000 7f700d47b700 1 -- 172.21.15.6:0/3685867731 shutdown_connections 2024-02-09T21:56:07.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.139+0000 7f700d47b700 1 --2- 172.21.15.6:0/3685867731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7008102800 0x7f70080fbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:07.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.139+0000 7f700d47b700 1 --2- 172.21.15.6:0/3685867731 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7008101e60 0x7f7008102230 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:07.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.139+0000 7f700d47b700 1 -- 172.21.15.6:0/3685867731 >> 172.21.15.6:0/3685867731 conn(0x7f70080f7a10 msgr2=0x7f70080f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:07.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.139+0000 7f700d47b700 1 -- 172.21.15.6:0/3685867731 shutdown_connections 2024-02-09T21:56:07.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.139+0000 7f700d47b700 1 -- 172.21.15.6:0/3685867731 wait complete. 2024-02-09T21:56:07.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.140+0000 7f700d47b700 1 Processor -- start 2024-02-09T21:56:07.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.140+0000 7f700d47b700 1 -- start start 2024-02-09T21:56:07.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.141+0000 7f700d47b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7008101e60 0x7f7008106d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:07.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.141+0000 7f700d47b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7008102800 0x7f7008107240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:07.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.141+0000 7f700d47b700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7008107840 con 0x7f7008101e60 2024-02-09T21:56:07.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.141+0000 7f700d47b700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70081079b0 con 0x7f7008102800 2024-02-09T21:56:07.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.141+0000 7f7006ffd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7008101e60 0x7f7008106d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:07.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.141+0000 7f7006ffd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7008101e60 0x7f7008106d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56834/0 (socket says 172.21.15.6:56834) 2024-02-09T21:56:07.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.141+0000 7f7006ffd700 1 -- 172.21.15.6:0/2970543687 learned_addr learned my addr 172.21.15.6:0/2970543687 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:56:07.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.141+0000 7f70067fc700 1 --2- 172.21.15.6:0/2970543687 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7008102800 0x7f7008107240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:07.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.141+0000 7f7006ffd700 1 -- 172.21.15.6:0/2970543687 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7008102800 msgr2=0x7f7008107240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:07.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.141+0000 7f7006ffd700 1 --2- 172.21.15.6:0/2970543687 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7008102800 0x7f7008107240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:07.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.141+0000 7f7006ffd700 1 -- 172.21.15.6:0/2970543687 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff8005040 con 0x7f7008101e60 2024-02-09T21:56:07.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.141+0000 7f70067fc700 1 --2- 172.21.15.6:0/2970543687 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7008102800 0x7f7008107240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T21:56:07.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.141+0000 7f7006ffd700 1 --2- 172.21.15.6:0/2970543687 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7008101e60 0x7f7008106d00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8009100 tx=0x7f6ff8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:07.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.141+0000 7f6fef7fe700 1 -- 172.21.15.6:0/2970543687 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ff800e040 con 0x7f7008101e60 2024-02-09T21:56:07.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.141+0000 7f6fef7fe700 1 -- 172.21.15.6:0/2970543687 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6ff8004290 con 0x7f7008101e60 2024-02-09T21:56:07.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.141+0000 7f700d47b700 1 -- 172.21.15.6:0/2970543687 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7008119cd0 con 0x7f7008101e60 2024-02-09T21:56:07.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.141+0000 7f6fef7fe700 1 -- 172.21.15.6:0/2970543687 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ff801c630 con 0x7f7008101e60 2024-02-09T21:56:07.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.141+0000 7f700d47b700 1 -- 172.21.15.6:0/2970543687 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f700811a130 con 0x7f7008101e60 2024-02-09T21:56:07.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.142+0000 7f700d47b700 1 -- 172.21.15.6:0/2970543687 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70080ff170 con 0x7f7008101e60 2024-02-09T21:56:07.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.143+0000 7f6fef7fe700 1 -- 172.21.15.6:0/2970543687 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f6ff800bbf0 con 0x7f7008101e60 2024-02-09T21:56:07.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.143+0000 7f6fef7fe700 1 --2- 172.21.15.6:0/2970543687 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f6ff00692a0 0x7f6ff006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:07.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.143+0000 7f6fef7fe700 1 -- 172.21.15.6:0/2970543687 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6ff80880d0 con 0x7f7008101e60 2024-02-09T21:56:07.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.143+0000 7f70067fc700 1 --2- 172.21.15.6:0/2970543687 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f6ff00692a0 0x7f6ff006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:07.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.144+0000 7f70067fc700 1 --2- 172.21.15.6:0/2970543687 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f6ff00692a0 0x7f6ff006b750 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7008108700 tx=0x7f6ffc009040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:07.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.147+0000 7f6fef7fe700 1 -- 172.21.15.6:0/2970543687 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6ff80547e0 con 0x7f7008101e60 2024-02-09T21:56:07.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:06 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/2951739961' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-09T21:56:07.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.342+0000 7f700d47b700 1 -- 172.21.15.6:0/2970543687 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f700804bed0 con 0x7f7008101e60 2024-02-09T21:56:07.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.343+0000 7f6fef7fe700 1 -- 172.21.15.6:0/2970543687 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f6ff8057e00 con 0x7f7008101e60 2024-02-09T21:56:07.347 INFO:teuthology.orchestra.run.smithi006.stdout:146028888081 2024-02-09T21:56:07.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.345+0000 7f700d47b700 1 -- 172.21.15.6:0/2970543687 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f6ff00692a0 msgr2=0x7f6ff006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:07.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.345+0000 7f700d47b700 1 --2- 172.21.15.6:0/2970543687 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f6ff00692a0 0x7f6ff006b750 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7008108700 tx=0x7f6ffc009040 comp rx=0 tx=0).stop 2024-02-09T21:56:07.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.345+0000 7f700d47b700 1 -- 172.21.15.6:0/2970543687 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7008101e60 msgr2=0x7f7008106d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:07.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.345+0000 7f700d47b700 1 --2- 172.21.15.6:0/2970543687 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7008101e60 0x7f7008106d00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8009100 tx=0x7f6ff8009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:07.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.345+0000 7f700d47b700 1 -- 172.21.15.6:0/2970543687 shutdown_connections 2024-02-09T21:56:07.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.345+0000 7f700d47b700 1 --2- 172.21.15.6:0/2970543687 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f6ff00692a0 0x7f6ff006b750 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:07.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.345+0000 7f700d47b700 1 --2- 172.21.15.6:0/2970543687 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7008101e60 0x7f7008106d00 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:07.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.345+0000 7f700d47b700 1 --2- 172.21.15.6:0/2970543687 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7008102800 0x7f7008107240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:07.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.345+0000 7f700d47b700 1 -- 172.21.15.6:0/2970543687 >> 172.21.15.6:0/2970543687 conn(0x7f70080f7a10 msgr2=0x7f70080f9400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:07.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.345+0000 7f700d47b700 1 -- 172.21.15.6:0/2970543687 shutdown_connections 2024-02-09T21:56:07.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:07.345+0000 7f700d47b700 1 -- 172.21.15.6:0/2970543687 wait complete. 2024-02-09T21:56:07.769 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149715 got 111669149717 for osd.3 2024-02-09T21:56:07.770 DEBUG:teuthology.parallel:result is None 2024-02-09T21:56:07.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.124+0000 7f860737f700 1 -- 172.21.15.6:0/4018817059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86000fe8f0 msgr2=0x7f8600106fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:07.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.124+0000 7f860737f700 1 --2- 172.21.15.6:0/4018817059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86000fe8f0 0x7f8600106fc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f85f0004940 tx=0x7f85f0009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:07.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.124+0000 7f860737f700 1 -- 172.21.15.6:0/4018817059 shutdown_connections 2024-02-09T21:56:07.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.124+0000 7f860737f700 1 --2- 172.21.15.6:0/4018817059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86000fe8f0 0x7f8600106fc0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:07.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.124+0000 7f860737f700 1 --2- 172.21.15.6:0/4018817059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86000fdf50 0x7f86000fe320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:07.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.124+0000 7f860737f700 1 -- 172.21.15.6:0/4018817059 >> 172.21.15.6:0/4018817059 conn(0x7f86000f7a10 msgr2=0x7f86000f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:07.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.124+0000 7f860737f700 1 -- 172.21.15.6:0/4018817059 shutdown_connections 2024-02-09T21:56:07.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.124+0000 7f860737f700 1 -- 172.21.15.6:0/4018817059 wait complete. 2024-02-09T21:56:07.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.125+0000 7f860737f700 1 Processor -- start 2024-02-09T21:56:07.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.125+0000 7f860737f700 1 -- start start 2024-02-09T21:56:07.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.125+0000 7f860737f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86000fdf50 0x7f8600113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:07.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.125+0000 7f860737f700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86000fe8f0 0x7f8600114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:07.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.125+0000 7f860737f700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8600114740 con 0x7f86000fe8f0 2024-02-09T21:56:07.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.125+0000 7f860737f700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860010d9e0 con 0x7f86000fdf50 2024-02-09T21:56:07.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.125+0000 7f860491a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86000fe8f0 0x7f8600114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:07.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.125+0000 7f860491a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86000fe8f0 0x7f8600114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56796/0 (socket says 172.21.15.6:56796) 2024-02-09T21:56:07.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.125+0000 7f860491a700 1 -- 172.21.15.6:0/2951739961 learned_addr learned my addr 172.21.15.6:0/2951739961 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:56:07.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.125+0000 7f860511b700 1 --2- 172.21.15.6:0/2951739961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86000fdf50 0x7f8600113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:07.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.125+0000 7f860491a700 1 -- 172.21.15.6:0/2951739961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86000fdf50 msgr2=0x7f8600113b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:07.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.125+0000 7f860491a700 1 --2- 172.21.15.6:0/2951739961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86000fdf50 0x7f8600113b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:07.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.125+0000 7f860491a700 1 -- 172.21.15.6:0/2951739961 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f0005040 con 0x7f86000fe8f0 2024-02-09T21:56:07.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.125+0000 7f860491a700 1 --2- 172.21.15.6:0/2951739961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86000fe8f0 0x7f8600114060 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f85f0004890 tx=0x7f85f0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:07.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.126+0000 7f85f5ffb700 1 -- 172.21.15.6:0/2951739961 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85f000e040 con 0x7f86000fe8f0 2024-02-09T21:56:07.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.126+0000 7f85f5ffb700 1 -- 172.21.15.6:0/2951739961 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f85f0004290 con 0x7f86000fe8f0 2024-02-09T21:56:07.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.126+0000 7f860737f700 1 -- 172.21.15.6:0/2951739961 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f860010dc60 con 0x7f86000fe8f0 2024-02-09T21:56:07.864 INFO:teuthology.orchestra.run.smithi006.stdout:167503724556 2024-02-09T21:56:07.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.126+0000 7f85f5ffb700 1 -- 172.21.15.6:0/2951739961 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85f001c630 con 0x7f86000fe8f0 2024-02-09T21:56:07.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.126+0000 7f860737f700 1 -- 172.21.15.6:0/2951739961 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f860010e1a0 con 0x7f86000fe8f0 2024-02-09T21:56:07.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.127+0000 7f860737f700 1 -- 172.21.15.6:0/2951739961 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8600104620 con 0x7f86000fe8f0 2024-02-09T21:56:07.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.127+0000 7f85f5ffb700 1 -- 172.21.15.6:0/2951739961 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f85f000bbf0 con 0x7f86000fe8f0 2024-02-09T21:56:07.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.128+0000 7f85f5ffb700 1 --2- 172.21.15.6:0/2951739961 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f85ec0692a0 0x7f85ec06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:07.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.128+0000 7f85f5ffb700 1 -- 172.21.15.6:0/2951739961 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f85f00881d0 con 0x7f86000fe8f0 2024-02-09T21:56:07.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.128+0000 7f860511b700 1 --2- 172.21.15.6:0/2951739961 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f85ec0692a0 0x7f85ec06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:07.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.128+0000 7f860511b700 1 --2- 172.21.15.6:0/2951739961 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f85ec0692a0 0x7f85ec06b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f85fc006fd0 tx=0x7f85fc008040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:07.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.132+0000 7f85f5ffb700 1 -- 172.21.15.6:0/2951739961 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f85f0058370 con 0x7f86000fe8f0 2024-02-09T21:56:07.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.326+0000 7f860737f700 1 -- 172.21.15.6:0/2951739961 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f86000fb610 con 0x7f86000fe8f0 2024-02-09T21:56:07.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.326+0000 7f85f5ffb700 1 -- 172.21.15.6:0/2951739961 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f85f0057f00 con 0x7f86000fe8f0 2024-02-09T21:56:07.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.328+0000 7f860737f700 1 -- 172.21.15.6:0/2951739961 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f85ec0692a0 msgr2=0x7f85ec06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:07.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.328+0000 7f860737f700 1 --2- 172.21.15.6:0/2951739961 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f85ec0692a0 0x7f85ec06b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f85fc006fd0 tx=0x7f85fc008040 comp rx=0 tx=0).stop 2024-02-09T21:56:07.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.328+0000 7f860737f700 1 -- 172.21.15.6:0/2951739961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86000fe8f0 msgr2=0x7f8600114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:07.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.328+0000 7f860737f700 1 --2- 172.21.15.6:0/2951739961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86000fe8f0 0x7f8600114060 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f85f0004890 tx=0x7f85f0009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:07.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.328+0000 7f860737f700 1 -- 172.21.15.6:0/2951739961 shutdown_connections 2024-02-09T21:56:07.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.328+0000 7f860737f700 1 --2- 172.21.15.6:0/2951739961 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f85ec0692a0 0x7f85ec06b750 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:07.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.328+0000 7f860737f700 1 --2- 172.21.15.6:0/2951739961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86000fdf50 0x7f8600113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:07.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.328+0000 7f860737f700 1 --2- 172.21.15.6:0/2951739961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86000fe8f0 0x7f8600114060 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:07.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.328+0000 7f860737f700 1 -- 172.21.15.6:0/2951739961 >> 172.21.15.6:0/2951739961 conn(0x7f86000f7a10 msgr2=0x7f86001017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:07.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.328+0000 7f860737f700 1 -- 172.21.15.6:0/2951739961 shutdown_connections 2024-02-09T21:56:07.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:06.328+0000 7f860737f700 1 -- 172.21.15.6:0/2951739961 wait complete. 2024-02-09T21:56:08.106 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:07 smithi006 ceph-mon[29258]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:08.106 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:07 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/2082415614' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-09T21:56:08.106 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:07 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/2970543687' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-09T21:56:08.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:07 smithi090 ceph-mon[36996]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:08.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:07 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/2082415614' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-09T21:56:08.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:07 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/2970543687' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-09T21:56:08.686 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724553 got 167503724556 for osd.5 2024-02-09T21:56:08.687 DEBUG:teuthology.parallel:result is None 2024-02-09T21:56:09.382 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888077 got 146028888081 for osd.4 2024-02-09T21:56:09.382 DEBUG:teuthology.parallel:result is None 2024-02-09T21:56:09.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.593+0000 7f90d453a700 1 -- 172.21.15.6:0/1867883660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90cc1038f0 msgr2=0x7f90cc0fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:09.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.593+0000 7f90d453a700 1 --2- 172.21.15.6:0/1867883660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90cc1038f0 0x7f90cc0fcf60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f90bc004940 tx=0x7f90bc009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:09.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.593+0000 7f90d453a700 1 -- 172.21.15.6:0/1867883660 shutdown_connections 2024-02-09T21:56:09.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.593+0000 7f90d453a700 1 --2- 172.21.15.6:0/1867883660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90cc1038f0 0x7f90cc0fcf60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:09.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.593+0000 7f90d453a700 1 --2- 172.21.15.6:0/1867883660 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90cc102f50 0x7f90cc103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:09.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.593+0000 7f90d453a700 1 -- 172.21.15.6:0/1867883660 >> 172.21.15.6:0/1867883660 conn(0x7f90cc0f8ae0 msgr2=0x7f90cc0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:09.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.594+0000 7f90d453a700 1 -- 172.21.15.6:0/1867883660 shutdown_connections 2024-02-09T21:56:09.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.594+0000 7f90d453a700 1 -- 172.21.15.6:0/1867883660 wait complete. 2024-02-09T21:56:09.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.594+0000 7f90d453a700 1 Processor -- start 2024-02-09T21:56:09.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.594+0000 7f90d453a700 1 -- start start 2024-02-09T21:56:09.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.594+0000 7f90d453a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90cc102f50 0x7f90cc070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:09.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.594+0000 7f90d453a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90cc1038f0 0x7f90cc0707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:09.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.594+0000 7f90d453a700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc070dd0 con 0x7f90cc1038f0 2024-02-09T21:56:09.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.594+0000 7f90d453a700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc070f40 con 0x7f90cc102f50 2024-02-09T21:56:09.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.594+0000 7f90d1ad5700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90cc1038f0 0x7f90cc0707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:09.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.594+0000 7f90d1ad5700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90cc1038f0 0x7f90cc0707d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52446/0 (socket says 172.21.15.6:52446) 2024-02-09T21:56:09.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.594+0000 7f90d1ad5700 1 -- 172.21.15.6:0/601289197 learned_addr learned my addr 172.21.15.6:0/601289197 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:56:09.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.595+0000 7f90d22d6700 1 --2- 172.21.15.6:0/601289197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90cc102f50 0x7f90cc070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:09.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.595+0000 7f90d1ad5700 1 -- 172.21.15.6:0/601289197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90cc102f50 msgr2=0x7f90cc070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:09.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.595+0000 7f90d1ad5700 1 --2- 172.21.15.6:0/601289197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90cc102f50 0x7f90cc070290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:09.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.595+0000 7f90d1ad5700 1 -- 172.21.15.6:0/601289197 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90bc005040 con 0x7f90cc1038f0 2024-02-09T21:56:09.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.595+0000 7f90d1ad5700 1 --2- 172.21.15.6:0/601289197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90cc1038f0 0x7f90cc0707d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f90bc0090d0 tx=0x7f90bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:09.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.595+0000 7f90c2ffd700 1 -- 172.21.15.6:0/601289197 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90bc00e040 con 0x7f90cc1038f0 2024-02-09T21:56:09.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.595+0000 7f90c2ffd700 1 -- 172.21.15.6:0/601289197 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f90bc00bbf0 con 0x7f90cc1038f0 2024-02-09T21:56:09.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.595+0000 7f90d453a700 1 -- 172.21.15.6:0/601289197 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90cc119cd0 con 0x7f90cc1038f0 2024-02-09T21:56:09.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.595+0000 7f90c2ffd700 1 -- 172.21.15.6:0/601289197 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90bc01c630 con 0x7f90cc1038f0 2024-02-09T21:56:09.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.595+0000 7f90d453a700 1 -- 172.21.15.6:0/601289197 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90cc11a130 con 0x7f90cc1038f0 2024-02-09T21:56:09.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.596+0000 7f90d453a700 1 -- 172.21.15.6:0/601289197 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90cc105420 con 0x7f90cc1038f0 2024-02-09T21:56:09.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.597+0000 7f90c2ffd700 1 -- 172.21.15.6:0/601289197 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f90bc00bd60 con 0x7f90cc1038f0 2024-02-09T21:56:09.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.597+0000 7f90c2ffd700 1 --2- 172.21.15.6:0/601289197 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f90b80692a0 0x7f90b806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:09.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.597+0000 7f90c2ffd700 1 -- 172.21.15.6:0/601289197 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f90bc0881f0 con 0x7f90cc1038f0 2024-02-09T21:56:09.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.597+0000 7f90d22d6700 1 --2- 172.21.15.6:0/601289197 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f90b80692a0 0x7f90b806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:09.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.598+0000 7f90d22d6700 1 --2- 172.21.15.6:0/601289197 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f90b80692a0 0x7f90b806b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f90c80041d0 tx=0x7f90c8009040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:09.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.602+0000 7f90c2ffd700 1 -- 172.21.15.6:0/601289197 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f90bc054860 con 0x7f90cc1038f0 2024-02-09T21:56:09.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.795+0000 7f90d453a700 1 -- 172.21.15.6:0/601289197 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f90cc071100 con 0x7f90cc1038f0 2024-02-09T21:56:09.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.796+0000 7f90c2ffd700 1 -- 172.21.15.6:0/601289197 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f90cc071100 con 0x7f90cc1038f0 2024-02-09T21:56:09.800 INFO:teuthology.orchestra.run.smithi006.stdout:81604378651 2024-02-09T21:56:09.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.798+0000 7f90d453a700 1 -- 172.21.15.6:0/601289197 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f90b80692a0 msgr2=0x7f90b806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:09.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.798+0000 7f90d453a700 1 --2- 172.21.15.6:0/601289197 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f90b80692a0 0x7f90b806b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f90c80041d0 tx=0x7f90c8009040 comp rx=0 tx=0).stop 2024-02-09T21:56:09.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.798+0000 7f90d453a700 1 -- 172.21.15.6:0/601289197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90cc1038f0 msgr2=0x7f90cc0707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:09.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.798+0000 7f90d453a700 1 --2- 172.21.15.6:0/601289197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90cc1038f0 0x7f90cc0707d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f90bc0090d0 tx=0x7f90bc009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:09.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.798+0000 7f90d453a700 1 -- 172.21.15.6:0/601289197 shutdown_connections 2024-02-09T21:56:09.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.798+0000 7f90d453a700 1 --2- 172.21.15.6:0/601289197 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f90b80692a0 0x7f90b806b750 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:09.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.798+0000 7f90d453a700 1 --2- 172.21.15.6:0/601289197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90cc102f50 0x7f90cc070290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:09.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.798+0000 7f90d453a700 1 --2- 172.21.15.6:0/601289197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90cc1038f0 0x7f90cc0707d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:09.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.798+0000 7f90d453a700 1 -- 172.21.15.6:0/601289197 >> 172.21.15.6:0/601289197 conn(0x7f90cc0f8ae0 msgr2=0x7f90cc0fa5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:09.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.798+0000 7f90d453a700 1 -- 172.21.15.6:0/601289197 shutdown_connections 2024-02-09T21:56:09.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:09.798+0000 7f90d453a700 1 -- 172.21.15.6:0/601289197 wait complete. 2024-02-09T21:56:10.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:09 smithi090 ceph-mon[36996]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:10.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:09 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/601289197' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-09T21:56:10.301 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:09 smithi006 ceph-mon[29258]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:10.301 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:09 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/601289197' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-09T21:56:10.326 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378649 got 81604378651 for osd.2 2024-02-09T21:56:10.326 DEBUG:teuthology.parallel:result is None 2024-02-09T21:56:10.326 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-09T21:56:10.327 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-09T21:56:10.585 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:56:12.069 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:11 smithi006 ceph-mon[29258]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:12.070 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:11 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:56:12.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:11 smithi090 ceph-mon[36996]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:12.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:11 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:56:12.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.291+0000 7fa7b50e2700 1 -- 172.21.15.6:0/3758585470 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7b0102f50 msgr2=0x7fa7b0103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:12.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.291+0000 7fa7b50e2700 1 --2- 172.21.15.6:0/3758585470 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7b0102f50 0x7fa7b0103320 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0009ce0 tx=0x7fa7a0004d10 comp rx=0 tx=0).stop 2024-02-09T21:56:12.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.291+0000 7fa7b50e2700 1 -- 172.21.15.6:0/3758585470 shutdown_connections 2024-02-09T21:56:12.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.291+0000 7fa7b50e2700 1 --2- 172.21.15.6:0/3758585470 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7b01038f0 0x7fa7b00fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:12.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.291+0000 7fa7b50e2700 1 --2- 172.21.15.6:0/3758585470 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7b0102f50 0x7fa7b0103320 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:12.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.291+0000 7fa7b50e2700 1 -- 172.21.15.6:0/3758585470 >> 172.21.15.6:0/3758585470 conn(0x7fa7b00f8ae0 msgr2=0x7fa7b00faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:12.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.291+0000 7fa7b50e2700 1 -- 172.21.15.6:0/3758585470 shutdown_connections 2024-02-09T21:56:12.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.291+0000 7fa7b50e2700 1 -- 172.21.15.6:0/3758585470 wait complete. 2024-02-09T21:56:12.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.292+0000 7fa7b50e2700 1 Processor -- start 2024-02-09T21:56:12.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.292+0000 7fa7b50e2700 1 -- start start 2024-02-09T21:56:12.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.292+0000 7fa7b50e2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7b0102f50 0x7fa7b0113be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:12.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.292+0000 7fa7b50e2700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7b01038f0 0x7fa7b0114120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:12.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.293+0000 7fa7b50e2700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b0114770 con 0x7fa7b01038f0 2024-02-09T21:56:12.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.293+0000 7fa7b50e2700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b010daa0 con 0x7fa7b0102f50 2024-02-09T21:56:12.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.293+0000 7fa7ae59c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7b01038f0 0x7fa7b0114120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:12.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.293+0000 7fa7ae59c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7b01038f0 0x7fa7b0114120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52466/0 (socket says 172.21.15.6:52466) 2024-02-09T21:56:12.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.293+0000 7fa7ae59c700 1 -- 172.21.15.6:0/3796070639 learned_addr learned my addr 172.21.15.6:0/3796070639 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:56:12.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.293+0000 7fa7aed9d700 1 --2- 172.21.15.6:0/3796070639 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7b0102f50 0x7fa7b0113be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:12.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.293+0000 7fa7ae59c700 1 -- 172.21.15.6:0/3796070639 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7b0102f50 msgr2=0x7fa7b0113be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:12.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.293+0000 7fa7ae59c700 1 --2- 172.21.15.6:0/3796070639 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7b0102f50 0x7fa7b0113be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:12.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.293+0000 7fa7ae59c700 1 -- 172.21.15.6:0/3796070639 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a0009940 con 0x7fa7b01038f0 2024-02-09T21:56:12.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.293+0000 7fa7ae59c700 1 --2- 172.21.15.6:0/3796070639 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7b01038f0 0x7fa7b0114120 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa798002a50 tx=0x7fa798004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:12.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.293+0000 7fa7a77fe700 1 -- 172.21.15.6:0/3796070639 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa798007cc0 con 0x7fa7b01038f0 2024-02-09T21:56:12.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.293+0000 7fa7b50e2700 1 -- 172.21.15.6:0/3796070639 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7b010dd80 con 0x7fa7b01038f0 2024-02-09T21:56:12.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.293+0000 7fa7a77fe700 1 -- 172.21.15.6:0/3796070639 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa79800f040 con 0x7fa7b01038f0 2024-02-09T21:56:12.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.293+0000 7fa7a77fe700 1 -- 172.21.15.6:0/3796070639 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa79800d9c0 con 0x7fa7b01038f0 2024-02-09T21:56:12.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.293+0000 7fa7b50e2700 1 -- 172.21.15.6:0/3796070639 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b010e2c0 con 0x7fa7b01038f0 2024-02-09T21:56:12.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.294+0000 7fa7b50e2700 1 -- 172.21.15.6:0/3796070639 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7b0100260 con 0x7fa7b01038f0 2024-02-09T21:56:12.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.295+0000 7fa7a77fe700 1 -- 172.21.15.6:0/3796070639 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fa798009730 con 0x7fa7b01038f0 2024-02-09T21:56:12.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.295+0000 7fa7a77fe700 1 --2- 172.21.15.6:0/3796070639 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fa79c0692a0 0x7fa79c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:12.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.295+0000 7fa7a77fe700 1 -- 172.21.15.6:0/3796070639 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa798086bf0 con 0x7fa7b01038f0 2024-02-09T21:56:12.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.295+0000 7fa7aed9d700 1 --2- 172.21.15.6:0/3796070639 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fa79c0692a0 0x7fa79c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:12.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.296+0000 7fa7aed9d700 1 --2- 172.21.15.6:0/3796070639 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fa79c0692a0 0x7fa79c06b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0003400 tx=0x7fa7a0004d10 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:12.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.300+0000 7fa7a77fe700 1 -- 172.21.15.6:0/3796070639 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa798056c70 con 0x7fa7b01038f0 2024-02-09T21:56:12.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.487+0000 7fa7b50e2700 1 -- 172.21.15.6:0/3796070639 --> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa7b01138b0 con 0x7fa79c0692a0 2024-02-09T21:56:12.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.488+0000 7fa7a77fe700 1 -- 172.21.15.6:0/3796070639 <== mgr.14216 v2:172.21.15.6:6800/2624137529 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27984 (secure 0 0 0) 0x7fa7b01138b0 con 0x7fa79c0692a0 2024-02-09T21:56:12.492 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:56:12.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.490+0000 7fa7b50e2700 1 -- 172.21.15.6:0/3796070639 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fa79c0692a0 msgr2=0x7fa79c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:12.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.490+0000 7fa7b50e2700 1 --2- 172.21.15.6:0/3796070639 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fa79c0692a0 0x7fa79c06b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0003400 tx=0x7fa7a0004d10 comp rx=0 tx=0).stop 2024-02-09T21:56:12.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.490+0000 7fa7b50e2700 1 -- 172.21.15.6:0/3796070639 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7b01038f0 msgr2=0x7fa7b0114120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:12.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.490+0000 7fa7b50e2700 1 --2- 172.21.15.6:0/3796070639 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7b01038f0 0x7fa7b0114120 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa798002a50 tx=0x7fa798004600 comp rx=0 tx=0).stop 2024-02-09T21:56:12.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.490+0000 7fa7b50e2700 1 -- 172.21.15.6:0/3796070639 shutdown_connections 2024-02-09T21:56:12.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.490+0000 7fa7b50e2700 1 --2- 172.21.15.6:0/3796070639 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fa79c0692a0 0x7fa79c06b750 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:12.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.490+0000 7fa7b50e2700 1 --2- 172.21.15.6:0/3796070639 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7b0102f50 0x7fa7b0113be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:12.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.490+0000 7fa7b50e2700 1 --2- 172.21.15.6:0/3796070639 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7b01038f0 0x7fa7b0114120 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:12.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.490+0000 7fa7b50e2700 1 -- 172.21.15.6:0/3796070639 >> 172.21.15.6:0/3796070639 conn(0x7fa7b00f8ae0 msgr2=0x7fa7b00fa4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:12.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.490+0000 7fa7b50e2700 1 -- 172.21.15.6:0/3796070639 shutdown_connections 2024-02-09T21:56:12.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:12.490+0000 7fa7b50e2700 1 -- 172.21.15.6:0/3796070639 wait complete. 2024-02-09T21:56:12.495 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-02-09T21:56:13.065 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2024-02-09T21:56:10.863824+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787600,"kb_used_data":3872,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683312,"statfs":{"total":575970213888,"available":574139711488,"internally_reserved":0,"allocated":3964928,"data_stored":2605601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002441"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":190,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-02-09T21:55:22.280530+0000","last_change":"2024-02-09T21:54:57.450689+0000","last_active":"2024-02-09T21:55:22.280530+0000","last_peered":"2024-02-09T21:55:22.280530+0000","last_clean":"2024-02-09T21:55:22.280530+0000","last_became_active":"2024-02-09T21:54:57.450382+0000","last_became_peered":"2024-02-09T21:54:57.450382+0000","last_unstale":"2024-02-09T21:55:22.280530+0000","last_undegraded":"2024-02-09T21:55:22.280530+0000","last_fullsized":"2024-02-09T21:55:22.280530+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T21:54:09.449399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T21:54:09.449399+0000","last_clean_scrub_stamp":"2024-02-09T21:54:09.449399+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:45:17.688149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]}]},{"osd":4,"up_from":34,"seq":146028888082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 21:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.55800000000000005}]},{"osd":1,"last update":"Fri Feb 9 21:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.51500000000000001}]},{"osd":2,"last update":"Fri Feb 9 21:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.59499999999999997}]},{"osd":3,"last update":"Fri Feb 9 21:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.53700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149718,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 21:55:32 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.69399999999999995}]},{"osd":1,"last update":"Fri Feb 9 21:55:32 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.58399999999999996}]},{"osd":2,"last update":"Fri Feb 9 21:55:32 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.61199999999999999}]},{"osd":4,"last update":"Fri Feb 9 21:55:58 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.623},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.63700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297640,"kb_used_data":352,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447512,"statfs":{"total":95995035648,"available":95690252288,"internally_reserved":0,"allocated":360448,"data_stored":138532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 21:55:08 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.25700000000000001}]},{"osd":1,"last update":"Fri Feb 9 21:55:08 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.26700000000000002}]},{"osd":3,"last update":"Fri Feb 9 21:55:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.64600000000000002}]},{"osd":4,"last update":"Fri Feb 9 21:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":8.2959999999999994,"5min":8.2959999999999994,"15min":8.2959999999999994},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.49099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 21:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.34200000000000003,"15min":0.33900000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.377,"15min":0.379},"min":{"1min":0.23100000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.50600000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.33100000000000002}]},{"osd":2,"last update":"Fri Feb 9 21:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.27700000000000002}]},{"osd":3,"last update":"Fri Feb 9 21:55:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.441}]},{"osd":4,"last update":"Fri Feb 9 21:55:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.45400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 21:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35499999999999998,"15min":0.35299999999999998},"min":{"1min":0.245,"5min":0.218,"15min":0.218},"max":{"1min":0.53200000000000003,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.37,"15min":0.36199999999999999},"min":{"1min":0.28100000000000003,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.27800000000000002}]},{"osd":2,"last update":"Fri Feb 9 21:55:13 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.31}]},{"osd":3,"last update":"Fri Feb 9 21:55:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.59099999999999997}]},{"osd":4,"last update":"Fri Feb 9 21:55:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.60499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T21:56:13.066 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-09T21:56:13.325 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:56:14.124 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:13 smithi006 ceph-mon[29258]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T21:56:14.125 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:13 smithi006 ceph-mon[29258]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:14.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:13 smithi090 ceph-mon[36996]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T21:56:14.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:13 smithi090 ceph-mon[36996]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:14.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.981+0000 7ff745497700 1 -- 172.21.15.6:0/4284419515 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff740103870 msgr2=0x7ff7400fcee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.981+0000 7ff745497700 1 --2- 172.21.15.6:0/4284419515 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff740103870 0x7ff7400fcee0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff728004940 tx=0x7ff728009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.981+0000 7ff745497700 1 -- 172.21.15.6:0/4284419515 shutdown_connections 2024-02-09T21:56:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.981+0000 7ff745497700 1 --2- 172.21.15.6:0/4284419515 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff740103870 0x7ff7400fcee0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.981+0000 7ff745497700 1 --2- 172.21.15.6:0/4284419515 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff740102ed0 0x7ff7401032a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.981+0000 7ff745497700 1 -- 172.21.15.6:0/4284419515 >> 172.21.15.6:0/4284419515 conn(0x7ff7400f8a40 msgr2=0x7ff7400fae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.981+0000 7ff745497700 1 -- 172.21.15.6:0/4284419515 shutdown_connections 2024-02-09T21:56:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.981+0000 7ff745497700 1 -- 172.21.15.6:0/4284419515 wait complete. 2024-02-09T21:56:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.982+0000 7ff745497700 1 Processor -- start 2024-02-09T21:56:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.982+0000 7ff745497700 1 -- start start 2024-02-09T21:56:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.982+0000 7ff745497700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff740102ed0 0x7ff7401139e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.982+0000 7ff745497700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff740103870 0x7ff740113f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.982+0000 7ff745497700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff740114570 con 0x7ff740103870 2024-02-09T21:56:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.982+0000 7ff745497700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff74010d8a0 con 0x7ff740102ed0 2024-02-09T21:56:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.982+0000 7ff73e7fc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff740103870 0x7ff740113f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.982+0000 7ff73e7fc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff740103870 0x7ff740113f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52484/0 (socket says 172.21.15.6:52484) 2024-02-09T21:56:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.982+0000 7ff73e7fc700 1 -- 172.21.15.6:0/1537991643 learned_addr learned my addr 172.21.15.6:0/1537991643 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:56:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.982+0000 7ff73effd700 1 --2- 172.21.15.6:0/1537991643 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff740102ed0 0x7ff7401139e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.983+0000 7ff73e7fc700 1 -- 172.21.15.6:0/1537991643 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff740102ed0 msgr2=0x7ff7401139e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.983+0000 7ff73e7fc700 1 --2- 172.21.15.6:0/1537991643 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff740102ed0 0x7ff7401139e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.983+0000 7ff73e7fc700 1 -- 172.21.15.6:0/1537991643 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff728005040 con 0x7ff740103870 2024-02-09T21:56:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.983+0000 7ff73e7fc700 1 --2- 172.21.15.6:0/1537991643 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff740103870 0x7ff740113f20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff7280090d0 tx=0x7ff728009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.983+0000 7ff7377fe700 1 -- 172.21.15.6:0/1537991643 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff72800e040 con 0x7ff740103870 2024-02-09T21:56:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.983+0000 7ff7377fe700 1 -- 172.21.15.6:0/1537991643 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff728009430 con 0x7ff740103870 2024-02-09T21:56:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.983+0000 7ff7377fe700 1 -- 172.21.15.6:0/1537991643 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff72801c3f0 con 0x7ff740103870 2024-02-09T21:56:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.983+0000 7ff745497700 1 -- 172.21.15.6:0/1537991643 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff74010db20 con 0x7ff740103870 2024-02-09T21:56:14.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.983+0000 7ff745497700 1 -- 172.21.15.6:0/1537991643 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff74010e0e0 con 0x7ff740103870 2024-02-09T21:56:14.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.984+0000 7ff745497700 1 -- 172.21.15.6:0/1537991643 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7401053a0 con 0x7ff740103870 2024-02-09T21:56:14.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.984+0000 7ff7377fe700 1 -- 172.21.15.6:0/1537991643 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7ff72800bbf0 con 0x7ff740103870 2024-02-09T21:56:14.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.985+0000 7ff7377fe700 1 --2- 172.21.15.6:0/1537991643 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7ff72c0692a0 0x7ff72c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:14.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.985+0000 7ff7377fe700 1 -- 172.21.15.6:0/1537991643 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff728088000 con 0x7ff740103870 2024-02-09T21:56:14.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.985+0000 7ff73effd700 1 --2- 172.21.15.6:0/1537991643 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7ff72c0692a0 0x7ff72c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:14.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.985+0000 7ff73effd700 1 --2- 172.21.15.6:0/1537991643 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7ff72c0692a0 0x7ff72c06b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff730007970 tx=0x7ff730009da0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:14.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:14.989+0000 7ff7377fe700 1 -- 172.21.15.6:0/1537991643 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff728058250 con 0x7ff740103870 2024-02-09T21:56:15.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:15.178+0000 7ff745497700 1 -- 172.21.15.6:0/1537991643 --> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff7401136b0 con 0x7ff72c0692a0 2024-02-09T21:56:15.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:15.180+0000 7ff7377fe700 1 -- 172.21.15.6:0/1537991643 <== mgr.14216 v2:172.21.15.6:6800/2624137529 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27872 (secure 0 0 0) 0x7ff7401136b0 con 0x7ff72c0692a0 2024-02-09T21:56:15.184 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:56:15.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:15.182+0000 7ff745497700 1 -- 172.21.15.6:0/1537991643 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7ff72c0692a0 msgr2=0x7ff72c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:15.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:15.182+0000 7ff745497700 1 --2- 172.21.15.6:0/1537991643 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7ff72c0692a0 0x7ff72c06b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff730007970 tx=0x7ff730009da0 comp rx=0 tx=0).stop 2024-02-09T21:56:15.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:15.182+0000 7ff745497700 1 -- 172.21.15.6:0/1537991643 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff740103870 msgr2=0x7ff740113f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:15.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:15.182+0000 7ff745497700 1 --2- 172.21.15.6:0/1537991643 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff740103870 0x7ff740113f20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff7280090d0 tx=0x7ff728009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:15.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:15.182+0000 7ff745497700 1 -- 172.21.15.6:0/1537991643 shutdown_connections 2024-02-09T21:56:15.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:15.182+0000 7ff745497700 1 --2- 172.21.15.6:0/1537991643 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7ff72c0692a0 0x7ff72c06b750 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:15.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:15.182+0000 7ff745497700 1 --2- 172.21.15.6:0/1537991643 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff740102ed0 0x7ff7401139e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:15.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:15.182+0000 7ff745497700 1 --2- 172.21.15.6:0/1537991643 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff740103870 0x7ff740113f20 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:15.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:15.182+0000 7ff745497700 1 -- 172.21.15.6:0/1537991643 >> 172.21.15.6:0/1537991643 conn(0x7ff7400f8a40 msgr2=0x7ff7400fa4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:15.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:15.182+0000 7ff745497700 1 -- 172.21.15.6:0/1537991643 shutdown_connections 2024-02-09T21:56:15.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:15.182+0000 7ff745497700 1 -- 172.21.15.6:0/1537991643 wait complete. 2024-02-09T21:56:15.186 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-02-09T21:56:15.815 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2024-02-09T21:56:14.864580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787600,"kb_used_data":3872,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683312,"statfs":{"total":575970213888,"available":574139711488,"internally_reserved":0,"allocated":3964928,"data_stored":2605601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002322"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":190,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-02-09T21:55:22.280530+0000","last_change":"2024-02-09T21:54:57.450689+0000","last_active":"2024-02-09T21:55:22.280530+0000","last_peered":"2024-02-09T21:55:22.280530+0000","last_clean":"2024-02-09T21:55:22.280530+0000","last_became_active":"2024-02-09T21:54:57.450382+0000","last_became_peered":"2024-02-09T21:54:57.450382+0000","last_unstale":"2024-02-09T21:55:22.280530+0000","last_undegraded":"2024-02-09T21:55:22.280530+0000","last_fullsized":"2024-02-09T21:55:22.280530+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T21:54:09.449399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T21:54:09.449399+0000","last_clean_scrub_stamp":"2024-02-09T21:54:09.449399+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:45:17.688149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724558,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]}]},{"osd":4,"up_from":34,"seq":146028888083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 21:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.54500000000000004}]},{"osd":1,"last update":"Fri Feb 9 21:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.504},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.58399999999999996}]},{"osd":2,"last update":"Fri Feb 9 21:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.51700000000000002}]},{"osd":3,"last update":"Fri Feb 9 21:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.378},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.40100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 21:55:32 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.48999999999999999}]},{"osd":1,"last update":"Fri Feb 9 21:55:32 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.47799999999999998}]},{"osd":2,"last update":"Fri Feb 9 21:55:32 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.46500000000000002}]},{"osd":4,"last update":"Fri Feb 9 21:55:58 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.503}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297640,"kb_used_data":352,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447512,"statfs":{"total":95995035648,"available":95690252288,"internally_reserved":0,"allocated":360448,"data_stored":138532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 21:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34599999999999997,"15min":0.34799999999999998},"min":{"1min":0.23799999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.504,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.254},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.34699999999999998,"15min":0.34899999999999998},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.47599999999999998,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.28699999999999998}]},{"osd":1,"last update":"Fri Feb 9 21:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.35399999999999998,"15min":0.35799999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.51900000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.374,"5min":0.39000000000000001,"15min":0.39300000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.56599999999999995,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.30399999999999999}]},{"osd":3,"last update":"Fri Feb 9 21:55:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.53200000000000003}]},{"osd":4,"last update":"Fri Feb 9 21:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":8.2959999999999994,"5min":8.2959999999999994,"15min":8.2959999999999994},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.51800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 21:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.34200000000000003,"15min":0.33900000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.377,"15min":0.379},"min":{"1min":0.23100000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.50600000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.55900000000000005}]},{"osd":2,"last update":"Fri Feb 9 21:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.495},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.60999999999999999}]},{"osd":3,"last update":"Fri Feb 9 21:55:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.748},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.76300000000000001}]},{"osd":4,"last update":"Fri Feb 9 21:55:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.67100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 21:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35499999999999998,"15min":0.35299999999999998},"min":{"1min":0.245,"5min":0.218,"15min":0.218},"max":{"1min":0.53200000000000003,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.37,"15min":0.36199999999999999},"min":{"1min":0.28100000000000003,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.25700000000000001}]},{"osd":2,"last update":"Fri Feb 9 21:55:13 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.30299999999999999}]},{"osd":3,"last update":"Fri Feb 9 21:55:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.37},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.38800000000000001}]},{"osd":4,"last update":"Fri Feb 9 21:55:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.40300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T21:56:15.816 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-09T21:56:15.816 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-09T21:56:15.816 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-09T21:56:15.816 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph health --format=json 2024-02-09T21:56:16.076 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:56:16.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:15 smithi006 ceph-mon[29258]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:16.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:15 smithi006 ceph-mon[29258]: from='client.14422 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T21:56:16.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:15 smithi090 ceph-mon[36996]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:16.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:15 smithi090 ceph-mon[36996]: from='client.14422 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T21:56:17.512 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:17 smithi006 ceph-mon[29258]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:17.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:17 smithi090 ceph-mon[36996]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:17.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.729+0000 7fe87b0fe700 1 -- 172.21.15.6:0/2441857246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe874102f40 msgr2=0x7fe874103310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:17.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.729+0000 7fe87b0fe700 1 --2- 172.21.15.6:0/2441857246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe874102f40 0x7fe874103310 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe868004940 tx=0x7fe868009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:17.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.729+0000 7fe87b0fe700 1 -- 172.21.15.6:0/2441857246 shutdown_connections 2024-02-09T21:56:17.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.729+0000 7fe87b0fe700 1 --2- 172.21.15.6:0/2441857246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8741038e0 0x7fe8740fcf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:17.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.729+0000 7fe87b0fe700 1 --2- 172.21.15.6:0/2441857246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe874102f40 0x7fe874103310 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:17.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.729+0000 7fe87b0fe700 1 -- 172.21.15.6:0/2441857246 >> 172.21.15.6:0/2441857246 conn(0x7fe8740f8af0 msgr2=0x7fe8740faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:17.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.729+0000 7fe87b0fe700 1 -- 172.21.15.6:0/2441857246 shutdown_connections 2024-02-09T21:56:17.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.729+0000 7fe87b0fe700 1 -- 172.21.15.6:0/2441857246 wait complete. 2024-02-09T21:56:17.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.730+0000 7fe87b0fe700 1 Processor -- start 2024-02-09T21:56:17.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.730+0000 7fe87b0fe700 1 -- start start 2024-02-09T21:56:17.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.730+0000 7fe87b0fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe874102f40 0x7fe874113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:17.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.730+0000 7fe87b0fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8741038e0 0x7fe874113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:17.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.730+0000 7fe87b0fe700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8741145f0 con 0x7fe874102f40 2024-02-09T21:56:17.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.730+0000 7fe87b0fe700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe87410d920 con 0x7fe8741038e0 2024-02-09T21:56:17.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.730+0000 7fe878e9a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe874102f40 0x7fe874113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:17.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.730+0000 7fe878e9a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe874102f40 0x7fe874113a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43190/0 (socket says 172.21.15.6:43190) 2024-02-09T21:56:17.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.730+0000 7fe878e9a700 1 -- 172.21.15.6:0/205482452 learned_addr learned my addr 172.21.15.6:0/205482452 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:56:17.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.731+0000 7fe873fff700 1 --2- 172.21.15.6:0/205482452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8741038e0 0x7fe874113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:17.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.731+0000 7fe878e9a700 1 -- 172.21.15.6:0/205482452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8741038e0 msgr2=0x7fe874113fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:17.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.731+0000 7fe878e9a700 1 --2- 172.21.15.6:0/205482452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8741038e0 0x7fe874113fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:17.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.731+0000 7fe878e9a700 1 -- 172.21.15.6:0/205482452 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe868005040 con 0x7fe874102f40 2024-02-09T21:56:17.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.731+0000 7fe873fff700 1 --2- 172.21.15.6:0/205482452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8741038e0 0x7fe874113fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T21:56:17.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.731+0000 7fe878e9a700 1 --2- 172.21.15.6:0/205482452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe874102f40 0x7fe874113a60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe868009fd0 tx=0x7fe868009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:17.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.731+0000 7fe8717fa700 1 -- 172.21.15.6:0/205482452 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe86800e040 con 0x7fe874102f40 2024-02-09T21:56:17.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.731+0000 7fe87b0fe700 1 -- 172.21.15.6:0/205482452 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe87410dba0 con 0x7fe874102f40 2024-02-09T21:56:17.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.731+0000 7fe8717fa700 1 -- 172.21.15.6:0/205482452 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe86800bbf0 con 0x7fe874102f40 2024-02-09T21:56:17.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.731+0000 7fe8717fa700 1 -- 172.21.15.6:0/205482452 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe86801c3f0 con 0x7fe874102f40 2024-02-09T21:56:17.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.731+0000 7fe87b0fe700 1 -- 172.21.15.6:0/205482452 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe87410e0e0 con 0x7fe874102f40 2024-02-09T21:56:17.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.732+0000 7fe87b0fe700 1 -- 172.21.15.6:0/205482452 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8740fc6f0 con 0x7fe874102f40 2024-02-09T21:56:17.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.733+0000 7fe8717fa700 1 -- 172.21.15.6:0/205482452 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fe86800bd60 con 0x7fe874102f40 2024-02-09T21:56:17.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.733+0000 7fe8717fa700 1 --2- 172.21.15.6:0/205482452 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fe8640692b0 0x7fe86406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:17.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.733+0000 7fe8717fa700 1 -- 172.21.15.6:0/205482452 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe868088190 con 0x7fe874102f40 2024-02-09T21:56:17.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.733+0000 7fe873fff700 1 --2- 172.21.15.6:0/205482452 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fe8640692b0 0x7fe86406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:17.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.733+0000 7fe873fff700 1 --2- 172.21.15.6:0/205482452 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fe8640692b0 0x7fe86406b760 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe860007fd0 tx=0x7fe860009040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:17.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.737+0000 7fe8717fa700 1 -- 172.21.15.6:0/205482452 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe868058290 con 0x7fe874102f40 2024-02-09T21:56:17.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.973+0000 7fe87b0fe700 1 -- 172.21.15.6:0/205482452 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe87410f040 con 0x7fe874102f40 2024-02-09T21:56:17.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.973+0000 7fe8717fa700 1 -- 172.21.15.6:0/205482452 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fe868057e20 con 0x7fe874102f40 2024-02-09T21:56:17.980 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:56:17.980 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-09T21:56:17.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.975+0000 7fe87b0fe700 1 -- 172.21.15.6:0/205482452 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fe8640692b0 msgr2=0x7fe86406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:17.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.975+0000 7fe87b0fe700 1 --2- 172.21.15.6:0/205482452 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fe8640692b0 0x7fe86406b760 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe860007fd0 tx=0x7fe860009040 comp rx=0 tx=0).stop 2024-02-09T21:56:17.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.975+0000 7fe87b0fe700 1 -- 172.21.15.6:0/205482452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe874102f40 msgr2=0x7fe874113a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:17.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.975+0000 7fe87b0fe700 1 --2- 172.21.15.6:0/205482452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe874102f40 0x7fe874113a60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe868009fd0 tx=0x7fe868009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:17.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.975+0000 7fe87b0fe700 1 -- 172.21.15.6:0/205482452 shutdown_connections 2024-02-09T21:56:17.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.975+0000 7fe87b0fe700 1 --2- 172.21.15.6:0/205482452 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fe8640692b0 0x7fe86406b760 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:17.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.975+0000 7fe87b0fe700 1 --2- 172.21.15.6:0/205482452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe874102f40 0x7fe874113a60 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:17.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.975+0000 7fe87b0fe700 1 --2- 172.21.15.6:0/205482452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8741038e0 0x7fe874113fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:17.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.975+0000 7fe87b0fe700 1 -- 172.21.15.6:0/205482452 >> 172.21.15.6:0/205482452 conn(0x7fe8740f8af0 msgr2=0x7fe8740fa4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:17.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.975+0000 7fe87b0fe700 1 -- 172.21.15.6:0/205482452 shutdown_connections 2024-02-09T21:56:17.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:17.975+0000 7fe87b0fe700 1 -- 172.21.15.6:0/205482452 wait complete. 2024-02-09T21:56:18.554 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:18 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/205482452' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-09T21:56:18.579 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-09T21:56:18.579 INFO:tasks.cephadm:Setup complete, yielding 2024-02-09T21:56:18.579 INFO:teuthology.run_tasks:Running task print... 2024-02-09T21:56:18.593 INFO:teuthology.task.print:**** done end installing quincy cephadm ... 2024-02-09T21:56:18.593 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-09T21:56:18.642 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi006.front.sepia.ceph.com 2024-02-09T21:56:18.642 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-02-09T21:56:18.704 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:18 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/205482452' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-09T21:56:18.909 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:56:19.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:19 smithi006 ceph-mon[29258]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:19.704 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:19 smithi090 ceph-mon[36996]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:20.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.522+0000 7f671fc60700 1 -- 172.21.15.6:0/1853829244 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67180fe0e0 msgr2=0x7f67180fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:20.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.522+0000 7f671fc60700 1 --2- 172.21.15.6:0/1853829244 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67180fe0e0 0x7f67180fe550 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f670c004940 tx=0x7f670c009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:20.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.522+0000 7f671fc60700 1 -- 172.21.15.6:0/1853829244 shutdown_connections 2024-02-09T21:56:20.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.522+0000 7f671fc60700 1 --2- 172.21.15.6:0/1853829244 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67180fe0e0 0x7f67180fe550 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:20.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.522+0000 7f671fc60700 1 --2- 172.21.15.6:0/1853829244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67180fcef0 0x7f67180fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:20.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.522+0000 7f671fc60700 1 -- 172.21.15.6:0/1853829244 >> 172.21.15.6:0/1853829244 conn(0x7f67180f8620 msgr2=0x7f67180faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:20.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.522+0000 7f671fc60700 1 -- 172.21.15.6:0/1853829244 shutdown_connections 2024-02-09T21:56:20.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.522+0000 7f671fc60700 1 -- 172.21.15.6:0/1853829244 wait complete. 2024-02-09T21:56:20.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.523+0000 7f671fc60700 1 Processor -- start 2024-02-09T21:56:20.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.523+0000 7f671fc60700 1 -- start start 2024-02-09T21:56:20.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.523+0000 7f671fc60700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67180fcef0 0x7f671810d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:20.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.523+0000 7f671fc60700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67180fe0e0 0x7f671810ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:20.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.523+0000 7f671fc60700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671810e560 con 0x7f67180fe0e0 2024-02-09T21:56:20.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.523+0000 7f671fc60700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671810eda0 con 0x7f67180fcef0 2024-02-09T21:56:20.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.523+0000 7f671d1fb700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67180fe0e0 0x7f671810ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:20.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.523+0000 7f671d1fb700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67180fe0e0 0x7f671810ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43204/0 (socket says 172.21.15.6:43204) 2024-02-09T21:56:20.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.523+0000 7f671d1fb700 1 -- 172.21.15.6:0/1398316044 learned_addr learned my addr 172.21.15.6:0/1398316044 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:56:20.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.523+0000 7f671d9fc700 1 --2- 172.21.15.6:0/1398316044 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67180fcef0 0x7f671810d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:20.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.523+0000 7f671d1fb700 1 -- 172.21.15.6:0/1398316044 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67180fcef0 msgr2=0x7f671810d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:20.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.523+0000 7f671d1fb700 1 --2- 172.21.15.6:0/1398316044 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67180fcef0 0x7f671810d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:20.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.523+0000 7f671d1fb700 1 -- 172.21.15.6:0/1398316044 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f670c005040 con 0x7f67180fe0e0 2024-02-09T21:56:20.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.523+0000 7f671d9fc700 1 --2- 172.21.15.6:0/1398316044 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67180fcef0 0x7f671810d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T21:56:20.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.524+0000 7f671d1fb700 1 --2- 172.21.15.6:0/1398316044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67180fe0e0 0x7f671810ded0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f670c009fd0 tx=0x7f670c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:20.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.524+0000 7f670a7fc700 1 -- 172.21.15.6:0/1398316044 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f670c00e040 con 0x7f67180fe0e0 2024-02-09T21:56:20.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.524+0000 7f670a7fc700 1 -- 172.21.15.6:0/1398316044 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f670c00bcc0 con 0x7f67180fe0e0 2024-02-09T21:56:20.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.524+0000 7f671fc60700 1 -- 172.21.15.6:0/1398316044 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f671810f020 con 0x7f67180fe0e0 2024-02-09T21:56:20.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.524+0000 7f670a7fc700 1 -- 172.21.15.6:0/1398316044 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f670c01c630 con 0x7f67180fe0e0 2024-02-09T21:56:20.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.524+0000 7f671fc60700 1 -- 172.21.15.6:0/1398316044 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f671810f560 con 0x7f67180fe0e0 2024-02-09T21:56:20.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.525+0000 7f671fc60700 1 -- 172.21.15.6:0/1398316044 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f671804bed0 con 0x7f67180fe0e0 2024-02-09T21:56:20.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.525+0000 7f670a7fc700 1 -- 172.21.15.6:0/1398316044 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f670c013980 con 0x7f67180fe0e0 2024-02-09T21:56:20.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.525+0000 7f670a7fc700 1 --2- 172.21.15.6:0/1398316044 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f67040692b0 0x7f670406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:20.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.526+0000 7f670a7fc700 1 -- 172.21.15.6:0/1398316044 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f670c088310 con 0x7f67180fe0e0 2024-02-09T21:56:20.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.526+0000 7f671d9fc700 1 --2- 172.21.15.6:0/1398316044 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f67040692b0 0x7f670406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:20.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.526+0000 7f671d9fc700 1 --2- 172.21.15.6:0/1398316044 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f67040692b0 0x7f670406b760 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f67180fcc30 tx=0x7f6714008040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:20.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.530+0000 7f670a7fc700 1 -- 172.21.15.6:0/1398316044 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f670c058410 con 0x7f67180fe0e0 2024-02-09T21:56:20.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.719+0000 7f671fc60700 1 -- 172.21.15.6:0/1398316044 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f671810f9d0 con 0x7f67180fe0e0 2024-02-09T21:56:20.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.724+0000 7f670a7fc700 1 -- 172.21.15.6:0/1398316044 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f670c057fa0 con 0x7f67180fe0e0 2024-02-09T21:56:20.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.726+0000 7f671fc60700 1 -- 172.21.15.6:0/1398316044 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f67040692b0 msgr2=0x7f670406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:20.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.726+0000 7f671fc60700 1 --2- 172.21.15.6:0/1398316044 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f67040692b0 0x7f670406b760 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f67180fcc30 tx=0x7f6714008040 comp rx=0 tx=0).stop 2024-02-09T21:56:20.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.726+0000 7f671fc60700 1 -- 172.21.15.6:0/1398316044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67180fe0e0 msgr2=0x7f671810ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:20.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.726+0000 7f671fc60700 1 --2- 172.21.15.6:0/1398316044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67180fe0e0 0x7f671810ded0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f670c009fd0 tx=0x7f670c009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:20.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.726+0000 7f671fc60700 1 -- 172.21.15.6:0/1398316044 shutdown_connections 2024-02-09T21:56:20.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.726+0000 7f671fc60700 1 --2- 172.21.15.6:0/1398316044 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f67040692b0 0x7f670406b760 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:20.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.726+0000 7f671fc60700 1 --2- 172.21.15.6:0/1398316044 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67180fcef0 0x7f671810d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:20.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.726+0000 7f671fc60700 1 --2- 172.21.15.6:0/1398316044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67180fe0e0 0x7f671810ded0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:20.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.726+0000 7f671fc60700 1 -- 172.21.15.6:0/1398316044 >> 172.21.15.6:0/1398316044 conn(0x7f67180f8620 msgr2=0x7f67181013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:20.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.726+0000 7f671fc60700 1 -- 172.21.15.6:0/1398316044 shutdown_connections 2024-02-09T21:56:20.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:20.726+0000 7f671fc60700 1 -- 172.21.15.6:0/1398316044 wait complete. 2024-02-09T21:56:21.442 INFO:teuthology.run_tasks:Running task print... 2024-02-09T21:56:21.452 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-02-09T21:56:21.452 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-09T21:56:21.491 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi006.front.sepia.ceph.com 2024-02-09T21:56:21.491 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- bash -c 'ceph orch status' 2024-02-09T21:56:21.751 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:56:22.063 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:21 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/1398316044' entity='client.admin' 2024-02-09T21:56:22.064 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:21 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:56:22.064 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:21 smithi006 ceph-mon[29258]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:22.064 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:21 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:56:22.064 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:21 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:56:22.065 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:21 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:22.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:21 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/1398316044' entity='client.admin' 2024-02-09T21:56:22.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:21 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:56:22.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:21 smithi090 ceph-mon[36996]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:22.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:21 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:56:22.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:21 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:56:22.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:21 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:23.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.351+0000 7f992db58700 1 -- 172.21.15.6:0/1374339468 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99280fb5e0 msgr2=0x7f99280fb9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:23.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.352+0000 7f992db58700 1 --2- 172.21.15.6:0/1374339468 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99280fb5e0 0x7f99280fb9f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f991c004940 tx=0x7f991c009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:23.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.352+0000 7f992db58700 1 -- 172.21.15.6:0/1374339468 shutdown_connections 2024-02-09T21:56:23.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.352+0000 7f992db58700 1 --2- 172.21.15.6:0/1374339468 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99280fc7f0 0x7f99280fcc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:23.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.352+0000 7f992db58700 1 --2- 172.21.15.6:0/1374339468 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99280fb5e0 0x7f99280fb9f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:23.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.352+0000 7f992db58700 1 -- 172.21.15.6:0/1374339468 >> 172.21.15.6:0/1374339468 conn(0x7f99280f6ca0 msgr2=0x7f99280f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:23.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.352+0000 7f992db58700 1 -- 172.21.15.6:0/1374339468 shutdown_connections 2024-02-09T21:56:23.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.352+0000 7f992db58700 1 -- 172.21.15.6:0/1374339468 wait complete. 2024-02-09T21:56:23.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.352+0000 7f992db58700 1 Processor -- start 2024-02-09T21:56:23.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.352+0000 7f992db58700 1 -- start start 2024-02-09T21:56:23.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.353+0000 7f992db58700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99280fb5e0 0x7f992810b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:23.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.353+0000 7f992db58700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99280fc7f0 0x7f992810bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:23.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.353+0000 7f992db58700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992810c380 con 0x7f99280fb5e0 2024-02-09T21:56:23.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.353+0000 7f992db58700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992810cc20 con 0x7f99280fc7f0 2024-02-09T21:56:23.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.353+0000 7f99277fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99280fb5e0 0x7f992810b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:23.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.353+0000 7f99277fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99280fb5e0 0x7f992810b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43218/0 (socket says 172.21.15.6:43218) 2024-02-09T21:56:23.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.353+0000 7f99277fe700 1 -- 172.21.15.6:0/503055146 learned_addr learned my addr 172.21.15.6:0/503055146 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:56:23.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.353+0000 7f99277fe700 1 -- 172.21.15.6:0/503055146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99280fc7f0 msgr2=0x7f992810bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:23.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.353+0000 7f99277fe700 1 --2- 172.21.15.6:0/503055146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99280fc7f0 0x7f992810bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:23.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.353+0000 7f99277fe700 1 -- 172.21.15.6:0/503055146 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f991c005040 con 0x7f99280fb5e0 2024-02-09T21:56:23.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.353+0000 7f99277fe700 1 --2- 172.21.15.6:0/503055146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99280fb5e0 0x7f992810b840 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f991c009150 tx=0x7f991c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:23.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.353+0000 7f990ffff700 1 -- 172.21.15.6:0/503055146 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f991c00e040 con 0x7f99280fb5e0 2024-02-09T21:56:23.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.353+0000 7f992db58700 1 -- 172.21.15.6:0/503055146 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f992810cea0 con 0x7f99280fb5e0 2024-02-09T21:56:23.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.353+0000 7f990ffff700 1 -- 172.21.15.6:0/503055146 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f991c00bbf0 con 0x7f99280fb5e0 2024-02-09T21:56:23.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.353+0000 7f990ffff700 1 -- 172.21.15.6:0/503055146 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f991c01c630 con 0x7f99280fb5e0 2024-02-09T21:56:23.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.353+0000 7f992db58700 1 -- 172.21.15.6:0/503055146 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f992810d3e0 con 0x7f99280fb5e0 2024-02-09T21:56:23.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.354+0000 7f992db58700 1 -- 172.21.15.6:0/503055146 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9928105880 con 0x7f99280fb5e0 2024-02-09T21:56:23.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.355+0000 7f990ffff700 1 -- 172.21.15.6:0/503055146 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f991c013420 con 0x7f99280fb5e0 2024-02-09T21:56:23.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.355+0000 7f990ffff700 1 --2- 172.21.15.6:0/503055146 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f99100692a0 0x7f991006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:23.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.355+0000 7f990ffff700 1 -- 172.21.15.6:0/503055146 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f991c088180 con 0x7f99280fb5e0 2024-02-09T21:56:23.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.355+0000 7f9926ffd700 1 --2- 172.21.15.6:0/503055146 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f99100692a0 0x7f991006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:23.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.355+0000 7f9926ffd700 1 --2- 172.21.15.6:0/503055146 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f99100692a0 0x7f991006b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f991800afd0 tx=0x7f991800adf0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:23.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.360+0000 7f990ffff700 1 -- 172.21.15.6:0/503055146 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f991c058280 con 0x7f99280fb5e0 2024-02-09T21:56:23.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.570+0000 7f992db58700 1 -- 172.21.15.6:0/503055146 --> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f992802b3b0 con 0x7f99100692a0 2024-02-09T21:56:23.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.571+0000 7f990ffff700 1 -- 172.21.15.6:0/503055146 <== mgr.14216 v2:172.21.15.6:6800/2624137529 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f992802b3b0 con 0x7f99100692a0 2024-02-09T21:56:23.576 INFO:teuthology.orchestra.run.smithi006.stdout:Backend: cephadm 2024-02-09T21:56:23.576 INFO:teuthology.orchestra.run.smithi006.stdout:Available: Yes 2024-02-09T21:56:23.576 INFO:teuthology.orchestra.run.smithi006.stdout:Paused: No 2024-02-09T21:56:23.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.573+0000 7f992db58700 1 -- 172.21.15.6:0/503055146 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f99100692a0 msgr2=0x7f991006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:23.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.573+0000 7f992db58700 1 --2- 172.21.15.6:0/503055146 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f99100692a0 0x7f991006b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f991800afd0 tx=0x7f991800adf0 comp rx=0 tx=0).stop 2024-02-09T21:56:23.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.573+0000 7f992db58700 1 -- 172.21.15.6:0/503055146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99280fb5e0 msgr2=0x7f992810b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:23.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.573+0000 7f992db58700 1 --2- 172.21.15.6:0/503055146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99280fb5e0 0x7f992810b840 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f991c009150 tx=0x7f991c009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:23.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.573+0000 7f992db58700 1 -- 172.21.15.6:0/503055146 shutdown_connections 2024-02-09T21:56:23.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.573+0000 7f992db58700 1 --2- 172.21.15.6:0/503055146 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f99100692a0 0x7f991006b750 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:23.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.573+0000 7f992db58700 1 --2- 172.21.15.6:0/503055146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99280fb5e0 0x7f992810b840 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:23.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.573+0000 7f992db58700 1 --2- 172.21.15.6:0/503055146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99280fc7f0 0x7f992810bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:23.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.573+0000 7f992db58700 1 -- 172.21.15.6:0/503055146 >> 172.21.15.6:0/503055146 conn(0x7f99280f6ca0 msgr2=0x7f99280f8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:23.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.573+0000 7f992db58700 1 -- 172.21.15.6:0/503055146 shutdown_connections 2024-02-09T21:56:23.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:23.573+0000 7f992db58700 1 -- 172.21.15.6:0/503055146 wait complete. 2024-02-09T21:56:24.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:23 smithi006 ceph-mon[29258]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:24.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:23 smithi090 ceph-mon[36996]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:24.207 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- bash -c 'ceph orch ps' 2024-02-09T21:56:24.466 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:56:25.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:24 smithi090 ceph-mon[36996]: from='client.14434 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:56:25.257 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:24 smithi006 ceph-mon[29258]: from='client.14434 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:56:26.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.042+0000 7f932d545700 1 -- 172.21.15.6:0/4045838208 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93280fc1c0 msgr2=0x7f93280fc630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:26.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.042+0000 7f932d545700 1 --2- 172.21.15.6:0/4045838208 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93280fc1c0 0x7f93280fc630 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f931c004940 tx=0x7f931c009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:26.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.042+0000 7f932d545700 1 -- 172.21.15.6:0/4045838208 shutdown_connections 2024-02-09T21:56:26.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.042+0000 7f932d545700 1 --2- 172.21.15.6:0/4045838208 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93280fc1c0 0x7f93280fc630 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:26.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.042+0000 7f932d545700 1 --2- 172.21.15.6:0/4045838208 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93280fb870 0x7f93280fbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:26.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.042+0000 7f932d545700 1 -- 172.21.15.6:0/4045838208 >> 172.21.15.6:0/4045838208 conn(0x7f93280f74e0 msgr2=0x7f93280f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:26.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.042+0000 7f932d545700 1 -- 172.21.15.6:0/4045838208 shutdown_connections 2024-02-09T21:56:26.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.043+0000 7f932d545700 1 -- 172.21.15.6:0/4045838208 wait complete. 2024-02-09T21:56:26.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.043+0000 7f932d545700 1 Processor -- start 2024-02-09T21:56:26.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.043+0000 7f932d545700 1 -- start start 2024-02-09T21:56:26.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.043+0000 7f932d545700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93280fb870 0x7f932810fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:26.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.043+0000 7f932d545700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93280fc1c0 0x7f93281100a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:26.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.043+0000 7f932d545700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93281106a0 con 0x7f93280fc1c0 2024-02-09T21:56:26.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.043+0000 7f932d545700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9328110f40 con 0x7f93280fb870 2024-02-09T21:56:26.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.043+0000 7f93267fc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93280fc1c0 0x7f93281100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:26.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.043+0000 7f93267fc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93280fc1c0 0x7f93281100a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43236/0 (socket says 172.21.15.6:43236) 2024-02-09T21:56:26.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.043+0000 7f93267fc700 1 -- 172.21.15.6:0/796149554 learned_addr learned my addr 172.21.15.6:0/796149554 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:56:26.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.044+0000 7f9326ffd700 1 --2- 172.21.15.6:0/796149554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93280fb870 0x7f932810fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:26.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.044+0000 7f93267fc700 1 -- 172.21.15.6:0/796149554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93280fb870 msgr2=0x7f932810fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:26.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.044+0000 7f93267fc700 1 --2- 172.21.15.6:0/796149554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93280fb870 0x7f932810fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:26.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.044+0000 7f93267fc700 1 -- 172.21.15.6:0/796149554 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f931c005040 con 0x7f93280fc1c0 2024-02-09T21:56:26.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.044+0000 7f93267fc700 1 --2- 172.21.15.6:0/796149554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93280fc1c0 0x7f93281100a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f931c0093b0 tx=0x7f931c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:26.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.044+0000 7f930f7fe700 1 -- 172.21.15.6:0/796149554 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f931c00e040 con 0x7f93280fc1c0 2024-02-09T21:56:26.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.044+0000 7f932d545700 1 -- 172.21.15.6:0/796149554 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93281111c0 con 0x7f93280fc1c0 2024-02-09T21:56:26.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.044+0000 7f930f7fe700 1 -- 172.21.15.6:0/796149554 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f931c00bcc0 con 0x7f93280fc1c0 2024-02-09T21:56:26.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.044+0000 7f930f7fe700 1 -- 172.21.15.6:0/796149554 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f931c01c630 con 0x7f93280fc1c0 2024-02-09T21:56:26.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.044+0000 7f932d545700 1 -- 172.21.15.6:0/796149554 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9328111700 con 0x7f93280fc1c0 2024-02-09T21:56:26.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.045+0000 7f932d545700 1 -- 172.21.15.6:0/796149554 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f932804bed0 con 0x7f93280fc1c0 2024-02-09T21:56:26.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.046+0000 7f930f7fe700 1 -- 172.21.15.6:0/796149554 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f931c013980 con 0x7f93280fc1c0 2024-02-09T21:56:26.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.046+0000 7f930f7fe700 1 --2- 172.21.15.6:0/796149554 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f93100692a0 0x7f931006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:26.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.046+0000 7f930f7fe700 1 -- 172.21.15.6:0/796149554 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f931c0881e0 con 0x7f93280fc1c0 2024-02-09T21:56:26.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.046+0000 7f9326ffd700 1 --2- 172.21.15.6:0/796149554 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f93100692a0 0x7f931006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:26.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.046+0000 7f9326ffd700 1 --2- 172.21.15.6:0/796149554 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f93100692a0 0x7f931006b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9318004f40 tx=0x7f931800b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:26.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.050+0000 7f930f7fe700 1 -- 172.21.15.6:0/796149554 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f931c0582e0 con 0x7f93280fc1c0 2024-02-09T21:56:26.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:25 smithi006 ceph-mon[29258]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:26.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:25 smithi090 ceph-mon[36996]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:26.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.253+0000 7f932d545700 1 -- 172.21.15.6:0/796149554 --> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9328111b50 con 0x7f93100692a0 2024-02-09T21:56:26.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.262+0000 7f930f7fe700 1 -- 172.21.15.6:0/796149554 <== mgr.14216 v2:172.21.15.6:6800/2624137529 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2682 (secure 0 0 0) 0x7f9328111b50 con 0x7f93100692a0 2024-02-09T21:56:26.265 INFO:teuthology.orchestra.run.smithi006.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-09T21:56:26.266 INFO:teuthology.orchestra.run.smithi006.stdout:alertmanager.smithi006 smithi006 *:9093,9094 running (3m) 2m ago 5m 26.4M - 0.25.0 c8568f914cd2 5c39f25f512b 2024-02-09T21:56:26.266 INFO:teuthology.orchestra.run.smithi006.stdout:crash.smithi006 smithi006 running (5m) 2m ago 5m 6665k - 17.2.7-505-g72ce7c1d a2a0bd126c96 821e62ef6577 2024-02-09T21:56:26.266 INFO:teuthology.orchestra.run.smithi006.stdout:crash.smithi090 smithi090 running (4m) 65s ago 4m 6665k - 17.2.7-505-g72ce7c1d a2a0bd126c96 c9a50a0d16b7 2024-02-09T21:56:26.266 INFO:teuthology.orchestra.run.smithi006.stdout:grafana.smithi006 smithi006 *:3000 running (3m) 2m ago 4m 78.0M - 9.4.7 954c08fa6188 32ff2ec358e5 2024-02-09T21:56:26.266 INFO:teuthology.orchestra.run.smithi006.stdout:mgr.smithi006.glirxt smithi006 *:9283 running (6m) 2m ago 6m 471M - 17.2.7-505-g72ce7c1d a2a0bd126c96 133e424caf5e 2024-02-09T21:56:26.266 INFO:teuthology.orchestra.run.smithi006.stdout:mgr.smithi090.csxosj smithi090 *:8443,9283 running (4m) 65s ago 4m 433M - 17.2.7-505-g72ce7c1d a2a0bd126c96 1c0eaba0034b 2024-02-09T21:56:26.266 INFO:teuthology.orchestra.run.smithi006.stdout:mon.smithi006 smithi006 running (6m) 2m ago 6m 41.5M 2048M 17.2.7-505-g72ce7c1d a2a0bd126c96 f8cba7495eab 2024-02-09T21:56:26.266 INFO:teuthology.orchestra.run.smithi006.stdout:mon.smithi090 smithi090 running (4m) 65s ago 4m 41.0M 2048M 17.2.7-505-g72ce7c1d a2a0bd126c96 4e4911a241f4 2024-02-09T21:56:26.266 INFO:teuthology.orchestra.run.smithi006.stdout:node-exporter.smithi006 smithi006 *:9100 running (5m) 2m ago 5m 16.8M - 1.5.0 0da6a335fe13 8c411b417300 2024-02-09T21:56:26.266 INFO:teuthology.orchestra.run.smithi006.stdout:node-exporter.smithi090 smithi090 *:9100 running (4m) 65s ago 4m 16.3M - 1.5.0 0da6a335fe13 274157586c39 2024-02-09T21:56:26.266 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0 smithi006 running (3m) 2m ago 3m 37.9M 4289M 17.2.7-505-g72ce7c1d a2a0bd126c96 1aea34e55f65 2024-02-09T21:56:26.266 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1 smithi006 running (2m) 2m ago 2m 35.1M 4289M 17.2.7-505-g72ce7c1d a2a0bd126c96 58f54ee70423 2024-02-09T21:56:26.266 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2 smithi006 running (2m) 2m ago 2m 11.0M 4289M 17.2.7-505-g72ce7c1d a2a0bd126c96 9dfe76e205a6 2024-02-09T21:56:26.266 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3 smithi090 running (2m) 65s ago 2m 42.4M 5312M 17.2.7-505-g72ce7c1d a2a0bd126c96 7cadb80e5790 2024-02-09T21:56:26.267 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4 smithi090 running (95s) 65s ago 95s 37.2M 5312M 17.2.7-505-g72ce7c1d a2a0bd126c96 1e173e0c2c75 2024-02-09T21:56:26.267 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5 smithi090 running (70s) 65s ago 70s 13.2M 5312M 17.2.7-505-g72ce7c1d a2a0bd126c96 bc873d1a2623 2024-02-09T21:56:26.267 INFO:teuthology.orchestra.run.smithi006.stdout:prometheus.smithi006 smithi006 *:9095 running (3m) 2m ago 4m 37.4M - 2.43.0 a07b618ecd1d ac6cb4bbb2d2 2024-02-09T21:56:26.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.264+0000 7f932d545700 1 -- 172.21.15.6:0/796149554 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f93100692a0 msgr2=0x7f931006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:26.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.264+0000 7f932d545700 1 --2- 172.21.15.6:0/796149554 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f93100692a0 0x7f931006b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9318004f40 tx=0x7f931800b040 comp rx=0 tx=0).stop 2024-02-09T21:56:26.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.264+0000 7f932d545700 1 -- 172.21.15.6:0/796149554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93280fc1c0 msgr2=0x7f93281100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:26.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.264+0000 7f932d545700 1 --2- 172.21.15.6:0/796149554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93280fc1c0 0x7f93281100a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f931c0093b0 tx=0x7f931c009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:26.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.264+0000 7f932d545700 1 -- 172.21.15.6:0/796149554 shutdown_connections 2024-02-09T21:56:26.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.264+0000 7f932d545700 1 --2- 172.21.15.6:0/796149554 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f93100692a0 0x7f931006b750 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:26.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.264+0000 7f932d545700 1 --2- 172.21.15.6:0/796149554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93280fb870 0x7f932810fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:26.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.264+0000 7f932d545700 1 --2- 172.21.15.6:0/796149554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93280fc1c0 0x7f93281100a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:26.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.264+0000 7f932d545700 1 -- 172.21.15.6:0/796149554 >> 172.21.15.6:0/796149554 conn(0x7f93280f74e0 msgr2=0x7f9328103ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:26.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.264+0000 7f932d545700 1 -- 172.21.15.6:0/796149554 shutdown_connections 2024-02-09T21:56:26.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:26.264+0000 7f932d545700 1 -- 172.21.15.6:0/796149554 wait complete. 2024-02-09T21:56:26.832 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- bash -c 'ceph orch ls' 2024-02-09T21:56:27.094 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:56:27.525 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:27 smithi006 ceph-mon[29258]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:56:27.526 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:27 smithi006 ceph-mon[29258]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:27.526 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:27 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:56:27.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:27 smithi090 ceph-mon[36996]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:56:27.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:27 smithi090 ceph-mon[36996]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:27.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:27 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:56:28.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.596+0000 7f0a642e1700 1 -- 172.21.15.6:0/2507358102 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a5c0fcfb0 msgr2=0x7f0a5c0fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:28.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.596+0000 7f0a642e1700 1 --2- 172.21.15.6:0/2507358102 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a5c0fcfb0 0x7f0a5c0fd420 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0a50004940 tx=0x7f0a50009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:28.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.596+0000 7f0a642e1700 1 -- 172.21.15.6:0/2507358102 shutdown_connections 2024-02-09T21:56:28.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.596+0000 7f0a642e1700 1 --2- 172.21.15.6:0/2507358102 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a5c0fcfb0 0x7f0a5c0fd420 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:28.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.596+0000 7f0a642e1700 1 --2- 172.21.15.6:0/2507358102 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a5c0fbe40 0x7f0a5c0fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:28.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.596+0000 7f0a642e1700 1 -- 172.21.15.6:0/2507358102 >> 172.21.15.6:0/2507358102 conn(0x7f0a5c0f7520 msgr2=0x7f0a5c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:28.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.596+0000 7f0a642e1700 1 -- 172.21.15.6:0/2507358102 shutdown_connections 2024-02-09T21:56:28.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.596+0000 7f0a642e1700 1 -- 172.21.15.6:0/2507358102 wait complete. 2024-02-09T21:56:28.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.596+0000 7f0a642e1700 1 Processor -- start 2024-02-09T21:56:28.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.596+0000 7f0a642e1700 1 -- start start 2024-02-09T21:56:28.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.597+0000 7f0a642e1700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a5c0fbe40 0x7f0a5c112a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:28.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.597+0000 7f0a642e1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a5c0fcfb0 0x7f0a5c114fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:28.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.597+0000 7f0a642e1700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5c066be0 con 0x7f0a5c0fbe40 2024-02-09T21:56:28.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.597+0000 7f0a642e1700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5c066d20 con 0x7f0a5c0fcfb0 2024-02-09T21:56:28.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.597+0000 7f0a6207d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a5c0fbe40 0x7f0a5c112a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:28.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.597+0000 7f0a6207d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a5c0fbe40 0x7f0a5c112a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53614/0 (socket says 172.21.15.6:53614) 2024-02-09T21:56:28.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.597+0000 7f0a6207d700 1 -- 172.21.15.6:0/2166438086 learned_addr learned my addr 172.21.15.6:0/2166438086 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:56:28.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.597+0000 7f0a6187c700 1 --2- 172.21.15.6:0/2166438086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a5c0fcfb0 0x7f0a5c114fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:28.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.597+0000 7f0a6207d700 1 -- 172.21.15.6:0/2166438086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a5c0fcfb0 msgr2=0x7f0a5c114fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:28.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.597+0000 7f0a6207d700 1 --2- 172.21.15.6:0/2166438086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a5c0fcfb0 0x7f0a5c114fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:28.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.597+0000 7f0a6207d700 1 -- 172.21.15.6:0/2166438086 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a50005040 con 0x7f0a5c0fbe40 2024-02-09T21:56:28.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.597+0000 7f0a6187c700 1 --2- 172.21.15.6:0/2166438086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a5c0fcfb0 0x7f0a5c114fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T21:56:28.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.597+0000 7f0a6207d700 1 --2- 172.21.15.6:0/2166438086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a5c0fbe40 0x7f0a5c112a60 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c001d80 tx=0x7f0a4c009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:28.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.597+0000 7f0a5affd700 1 -- 172.21.15.6:0/2166438086 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a4c007d10 con 0x7f0a5c0fbe40 2024-02-09T21:56:28.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.597+0000 7f0a642e1700 1 -- 172.21.15.6:0/2166438086 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a5c1154f0 con 0x7f0a5c0fbe40 2024-02-09T21:56:28.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.597+0000 7f0a5affd700 1 -- 172.21.15.6:0/2166438086 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a4c007e70 con 0x7f0a5c0fbe40 2024-02-09T21:56:28.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.597+0000 7f0a5affd700 1 -- 172.21.15.6:0/2166438086 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a4c014640 con 0x7f0a5c0fbe40 2024-02-09T21:56:28.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.598+0000 7f0a642e1700 1 -- 172.21.15.6:0/2166438086 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a5c1158f0 con 0x7f0a5c0fbe40 2024-02-09T21:56:28.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.598+0000 7f0a642e1700 1 -- 172.21.15.6:0/2166438086 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a5c04bed0 con 0x7f0a5c0fbe40 2024-02-09T21:56:28.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.599+0000 7f0a5affd700 1 -- 172.21.15.6:0/2166438086 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f0a4c00e960 con 0x7f0a5c0fbe40 2024-02-09T21:56:28.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.599+0000 7f0a5affd700 1 --2- 172.21.15.6:0/2166438086 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f0a440692b0 0x7f0a4406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:28.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.599+0000 7f0a6187c700 1 --2- 172.21.15.6:0/2166438086 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f0a440692b0 0x7f0a4406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:28.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.599+0000 7f0a5affd700 1 -- 172.21.15.6:0/2166438086 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0a4c086c30 con 0x7f0a5c0fbe40 2024-02-09T21:56:28.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.600+0000 7f0a6187c700 1 --2- 172.21.15.6:0/2166438086 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f0a440692b0 0x7f0a4406b760 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0a5000b7a0 tx=0x7f0a50009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:28.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.604+0000 7f0a5affd700 1 -- 172.21.15.6:0/2166438086 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0a4c0532d0 con 0x7f0a5c0fbe40 2024-02-09T21:56:28.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.804+0000 7f0a642e1700 1 -- 172.21.15.6:0/2166438086 --> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a5c101990 con 0x7f0a440692b0 2024-02-09T21:56:28.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.809+0000 7f0a5affd700 1 -- 172.21.15.6:0/2166438086 <== mgr.14216 v2:172.21.15.6:6800/2624137529 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1152 (secure 0 0 0) 0x7f0a5c101990 con 0x7f0a440692b0 2024-02-09T21:56:28.812 INFO:teuthology.orchestra.run.smithi006.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-02-09T21:56:28.812 INFO:teuthology.orchestra.run.smithi006.stdout:alertmanager ?:9093,9094 1/1 2m ago 5m count:1 2024-02-09T21:56:28.813 INFO:teuthology.orchestra.run.smithi006.stdout:crash 2/2 2m ago 6m * 2024-02-09T21:56:28.813 INFO:teuthology.orchestra.run.smithi006.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-02-09T21:56:28.813 INFO:teuthology.orchestra.run.smithi006.stdout:mgr 2/2 2m ago 6m count:2 2024-02-09T21:56:28.813 INFO:teuthology.orchestra.run.smithi006.stdout:mon 2/2 2m ago 5m smithi006:172.21.15.6=smithi006;smithi090:172.21.15.90=smithi090;count:2 2024-02-09T21:56:28.813 INFO:teuthology.orchestra.run.smithi006.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-02-09T21:56:28.813 INFO:teuthology.orchestra.run.smithi006.stdout:osd 6 2m ago - 2024-02-09T21:56:28.813 INFO:teuthology.orchestra.run.smithi006.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-02-09T21:56:28.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.810+0000 7f0a642e1700 1 -- 172.21.15.6:0/2166438086 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f0a440692b0 msgr2=0x7f0a4406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:28.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.810+0000 7f0a642e1700 1 --2- 172.21.15.6:0/2166438086 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f0a440692b0 0x7f0a4406b760 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0a5000b7a0 tx=0x7f0a50009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:28.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.811+0000 7f0a642e1700 1 -- 172.21.15.6:0/2166438086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a5c0fbe40 msgr2=0x7f0a5c112a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:28.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.811+0000 7f0a642e1700 1 --2- 172.21.15.6:0/2166438086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a5c0fbe40 0x7f0a5c112a60 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c001d80 tx=0x7f0a4c009ec0 comp rx=0 tx=0).stop 2024-02-09T21:56:28.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.811+0000 7f0a642e1700 1 -- 172.21.15.6:0/2166438086 shutdown_connections 2024-02-09T21:56:28.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.811+0000 7f0a642e1700 1 --2- 172.21.15.6:0/2166438086 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f0a440692b0 0x7f0a4406b760 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:28.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.811+0000 7f0a642e1700 1 --2- 172.21.15.6:0/2166438086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a5c0fbe40 0x7f0a5c112a60 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:28.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.811+0000 7f0a642e1700 1 --2- 172.21.15.6:0/2166438086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a5c0fcfb0 0x7f0a5c114fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:28.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.811+0000 7f0a642e1700 1 -- 172.21.15.6:0/2166438086 >> 172.21.15.6:0/2166438086 conn(0x7f0a5c0f7520 msgr2=0x7f0a5c100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:28.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.811+0000 7f0a642e1700 1 -- 172.21.15.6:0/2166438086 shutdown_connections 2024-02-09T21:56:28.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:28.811+0000 7f0a642e1700 1 -- 172.21.15.6:0/2166438086 wait complete. 2024-02-09T21:56:29.562 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- bash -c 'ceph orch host ls' 2024-02-09T21:56:29.825 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:56:30.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:29 smithi090 ceph-mon[36996]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:56:30.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:29 smithi090 ceph-mon[36996]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:30.240 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:29 smithi006 ceph-mon[29258]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:56:30.240 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:29 smithi006 ceph-mon[29258]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:31.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.420+0000 7f7f8f424700 1 -- 172.21.15.6:0/4285710252 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f880fbe40 msgr2=0x7f7f880fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:31.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.420+0000 7f7f8f424700 1 --2- 172.21.15.6:0/4285710252 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f880fbe40 0x7f7f880fc250 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7f84004940 tx=0x7f7f84009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:31.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.420+0000 7f7f8f424700 1 -- 172.21.15.6:0/4285710252 shutdown_connections 2024-02-09T21:56:31.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.420+0000 7f7f8f424700 1 --2- 172.21.15.6:0/4285710252 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f880fcfb0 0x7f7f880fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:31.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.420+0000 7f7f8f424700 1 --2- 172.21.15.6:0/4285710252 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f880fbe40 0x7f7f880fc250 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:31.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.420+0000 7f7f8f424700 1 -- 172.21.15.6:0/4285710252 >> 172.21.15.6:0/4285710252 conn(0x7f7f880f74e0 msgr2=0x7f7f880f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:31.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.420+0000 7f7f8f424700 1 -- 172.21.15.6:0/4285710252 shutdown_connections 2024-02-09T21:56:31.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.420+0000 7f7f8f424700 1 -- 172.21.15.6:0/4285710252 wait complete. 2024-02-09T21:56:31.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.420+0000 7f7f8f424700 1 Processor -- start 2024-02-09T21:56:31.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.420+0000 7f7f8f424700 1 -- start start 2024-02-09T21:56:31.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.421+0000 7f7f8f424700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f880fbe40 0x7f7f880fff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.421+0000 7f7f8f424700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f880fcfb0 0x7f7f880fe380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.421+0000 7f7f8f424700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f880fe980 con 0x7f7f880fcfb0 2024-02-09T21:56:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.421+0000 7f7f8f424700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f880feaf0 con 0x7f7f880fbe40 2024-02-09T21:56:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.421+0000 7f7f8c9bf700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f880fcfb0 0x7f7f880fe380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.421+0000 7f7f8c9bf700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f880fcfb0 0x7f7f880fe380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53632/0 (socket says 172.21.15.6:53632) 2024-02-09T21:56:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.421+0000 7f7f8c9bf700 1 -- 172.21.15.6:0/1138598159 learned_addr learned my addr 172.21.15.6:0/1138598159 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:56:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.421+0000 7f7f8d1c0700 1 --2- 172.21.15.6:0/1138598159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f880fbe40 0x7f7f880fff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.421+0000 7f7f8c9bf700 1 -- 172.21.15.6:0/1138598159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f880fbe40 msgr2=0x7f7f880fff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.421+0000 7f7f8c9bf700 1 --2- 172.21.15.6:0/1138598159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f880fbe40 0x7f7f880fff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.421+0000 7f7f8c9bf700 1 -- 172.21.15.6:0/1138598159 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f84005040 con 0x7f7f880fcfb0 2024-02-09T21:56:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.421+0000 7f7f8d1c0700 1 --2- 172.21.15.6:0/1138598159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f880fbe40 0x7f7f880fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T21:56:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.421+0000 7f7f8c9bf700 1 --2- 172.21.15.6:0/1138598159 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f880fcfb0 0x7f7f880fe380 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7f7800c830 tx=0x7f7f78004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:31.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.421+0000 7f7f7dffb700 1 -- 172.21.15.6:0/1138598159 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f7800a760 con 0x7f7f880fcfb0 2024-02-09T21:56:31.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.421+0000 7f7f8f424700 1 -- 172.21.15.6:0/1138598159 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f880fedd0 con 0x7f7f880fcfb0 2024-02-09T21:56:31.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.421+0000 7f7f7dffb700 1 -- 172.21.15.6:0/1138598159 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f7800ad90 con 0x7f7f880fcfb0 2024-02-09T21:56:31.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.422+0000 7f7f7dffb700 1 -- 172.21.15.6:0/1138598159 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f780113a0 con 0x7f7f880fcfb0 2024-02-09T21:56:31.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.422+0000 7f7f8f424700 1 -- 172.21.15.6:0/1138598159 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f88115970 con 0x7f7f880fcfb0 2024-02-09T21:56:31.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.423+0000 7f7f8f424700 1 -- 172.21.15.6:0/1138598159 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f8804bed0 con 0x7f7f880fcfb0 2024-02-09T21:56:31.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.423+0000 7f7f7dffb700 1 -- 172.21.15.6:0/1138598159 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f7f78012400 con 0x7f7f880fcfb0 2024-02-09T21:56:31.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.423+0000 7f7f7dffb700 1 --2- 172.21.15.6:0/1138598159 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f7f740692a0 0x7f7f7406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:31.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.424+0000 7f7f7dffb700 1 -- 172.21.15.6:0/1138598159 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7f78086ef0 con 0x7f7f880fcfb0 2024-02-09T21:56:31.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.424+0000 7f7f8d1c0700 1 --2- 172.21.15.6:0/1138598159 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f7f740692a0 0x7f7f7406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:31.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.424+0000 7f7f8d1c0700 1 --2- 172.21.15.6:0/1138598159 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f7f740692a0 0x7f7f7406b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7f8400b3b0 tx=0x7f7f84009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:31.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.428+0000 7f7f7dffb700 1 -- 172.21.15.6:0/1138598159 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7f78056ff0 con 0x7f7f880fcfb0 2024-02-09T21:56:31.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.627+0000 7f7f8f424700 1 -- 172.21.15.6:0/1138598159 --> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f88065dd0 con 0x7f7f740692a0 2024-02-09T21:56:31.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.628+0000 7f7f7dffb700 1 -- 172.21.15.6:0/1138598159 <== mgr.14216 v2:172.21.15.6:6800/2624137529 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+145 (secure 0 0 0) 0x7f7f88065dd0 con 0x7f7f740692a0 2024-02-09T21:56:31.632 INFO:teuthology.orchestra.run.smithi006.stdout:HOST ADDR LABELS STATUS 2024-02-09T21:56:31.632 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006 172.21.15.6 2024-02-09T21:56:31.632 INFO:teuthology.orchestra.run.smithi006.stdout:smithi090 172.21.15.90 2024-02-09T21:56:31.632 INFO:teuthology.orchestra.run.smithi006.stdout:2 hosts in cluster 2024-02-09T21:56:31.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.630+0000 7f7f8f424700 1 -- 172.21.15.6:0/1138598159 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f7f740692a0 msgr2=0x7f7f7406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:31.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.630+0000 7f7f8f424700 1 --2- 172.21.15.6:0/1138598159 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f7f740692a0 0x7f7f7406b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7f8400b3b0 tx=0x7f7f84009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:31.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.630+0000 7f7f8f424700 1 -- 172.21.15.6:0/1138598159 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f880fcfb0 msgr2=0x7f7f880fe380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:31.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.630+0000 7f7f8f424700 1 --2- 172.21.15.6:0/1138598159 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f880fcfb0 0x7f7f880fe380 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7f7800c830 tx=0x7f7f78004470 comp rx=0 tx=0).stop 2024-02-09T21:56:31.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.630+0000 7f7f8f424700 1 -- 172.21.15.6:0/1138598159 shutdown_connections 2024-02-09T21:56:31.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.630+0000 7f7f8f424700 1 --2- 172.21.15.6:0/1138598159 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f7f740692a0 0x7f7f7406b750 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:31.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.630+0000 7f7f8f424700 1 --2- 172.21.15.6:0/1138598159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f880fbe40 0x7f7f880fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:31.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.630+0000 7f7f8f424700 1 --2- 172.21.15.6:0/1138598159 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f880fcfb0 0x7f7f880fe380 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:31.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.630+0000 7f7f8f424700 1 -- 172.21.15.6:0/1138598159 >> 172.21.15.6:0/1138598159 conn(0x7f7f880f74e0 msgr2=0x7f7f880656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:31.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.630+0000 7f7f8f424700 1 -- 172.21.15.6:0/1138598159 shutdown_connections 2024-02-09T21:56:31.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:31.630+0000 7f7f8f424700 1 -- 172.21.15.6:0/1138598159 wait complete. 2024-02-09T21:56:31.975 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:31 smithi090 ceph-mon[36996]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:32.217 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:31 smithi006 ceph-mon[29258]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:32.245 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- bash -c 'ceph orch device ls' 2024-02-09T21:56:32.506 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:56:33.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:32 smithi090 ceph-mon[36996]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:56:33.273 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:32 smithi006 ceph-mon[29258]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:56:34.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:33 smithi090 ceph-mon[36996]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:34.262 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:33 smithi006 ceph-mon[29258]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:34.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.277+0000 7f3e3930c700 1 -- 172.21.15.6:0/175218688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e340fda80 msgr2=0x7f3e340ffe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:34.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.277+0000 7f3e3930c700 1 --2- 172.21.15.6:0/175218688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e340fda80 0x7f3e340ffe60 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c004940 tx=0x7f3e2c009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:34.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.277+0000 7f3e3930c700 1 -- 172.21.15.6:0/175218688 shutdown_connections 2024-02-09T21:56:34.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.277+0000 7f3e3930c700 1 --2- 172.21.15.6:0/175218688 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e341003a0 0x7f3e34102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:34.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.277+0000 7f3e3930c700 1 --2- 172.21.15.6:0/175218688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e340fda80 0x7f3e340ffe60 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:34.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.277+0000 7f3e3930c700 1 -- 172.21.15.6:0/175218688 >> 172.21.15.6:0/175218688 conn(0x7f3e340f74e0 msgr2=0x7f3e340f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:34.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.277+0000 7f3e3930c700 1 -- 172.21.15.6:0/175218688 shutdown_connections 2024-02-09T21:56:34.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.277+0000 7f3e3930c700 1 -- 172.21.15.6:0/175218688 wait complete. 2024-02-09T21:56:34.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.278+0000 7f3e3930c700 1 Processor -- start 2024-02-09T21:56:34.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.278+0000 7f3e3930c700 1 -- start start 2024-02-09T21:56:34.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.278+0000 7f3e3930c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e340fda80 0x7f3e3410d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:34.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.278+0000 7f3e3930c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e341003a0 0x7f3e3410de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:34.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.278+0000 7f3e3930c700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3410e470 con 0x7f3e341003a0 2024-02-09T21:56:34.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.278+0000 7f3e3930c700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3410ed10 con 0x7f3e340fda80 2024-02-09T21:56:34.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.278+0000 7f3e327fc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e341003a0 0x7f3e3410de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:34.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.278+0000 7f3e327fc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e341003a0 0x7f3e3410de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53638/0 (socket says 172.21.15.6:53638) 2024-02-09T21:56:34.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.278+0000 7f3e327fc700 1 -- 172.21.15.6:0/956114880 learned_addr learned my addr 172.21.15.6:0/956114880 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:56:34.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.278+0000 7f3e32ffd700 1 --2- 172.21.15.6:0/956114880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e340fda80 0x7f3e3410d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:34.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.278+0000 7f3e327fc700 1 -- 172.21.15.6:0/956114880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e340fda80 msgr2=0x7f3e3410d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:34.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.278+0000 7f3e327fc700 1 --2- 172.21.15.6:0/956114880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e340fda80 0x7f3e3410d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:34.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.278+0000 7f3e327fc700 1 -- 172.21.15.6:0/956114880 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e2c005040 con 0x7f3e341003a0 2024-02-09T21:56:34.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.279+0000 7f3e327fc700 1 --2- 172.21.15.6:0/956114880 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e341003a0 0x7f3e3410de70 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c002a50 tx=0x7f3e1c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:34.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.279+0000 7f3e2b7fe700 1 -- 172.21.15.6:0/956114880 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e1c007cc0 con 0x7f3e341003a0 2024-02-09T21:56:34.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.279+0000 7f3e3930c700 1 -- 172.21.15.6:0/956114880 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e3410eff0 con 0x7f3e341003a0 2024-02-09T21:56:34.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.279+0000 7f3e2b7fe700 1 -- 172.21.15.6:0/956114880 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3e1c00f040 con 0x7f3e341003a0 2024-02-09T21:56:34.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.279+0000 7f3e2b7fe700 1 -- 172.21.15.6:0/956114880 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e1c00da30 con 0x7f3e341003a0 2024-02-09T21:56:34.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.279+0000 7f3e3930c700 1 -- 172.21.15.6:0/956114880 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e3410f530 con 0x7f3e341003a0 2024-02-09T21:56:34.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.280+0000 7f3e3930c700 1 -- 172.21.15.6:0/956114880 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e3404bed0 con 0x7f3e341003a0 2024-02-09T21:56:34.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.280+0000 7f3e2b7fe700 1 -- 172.21.15.6:0/956114880 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f3e1c00dbb0 con 0x7f3e341003a0 2024-02-09T21:56:34.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.281+0000 7f3e2b7fe700 1 --2- 172.21.15.6:0/956114880 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f3e200692a0 0x7f3e2006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:34.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.281+0000 7f3e2b7fe700 1 -- 172.21.15.6:0/956114880 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3e1c086c00 con 0x7f3e341003a0 2024-02-09T21:56:34.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.281+0000 7f3e32ffd700 1 --2- 172.21.15.6:0/956114880 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f3e200692a0 0x7f3e2006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:34.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.281+0000 7f3e32ffd700 1 --2- 172.21.15.6:0/956114880 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f3e200692a0 0x7f3e2006b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c004f90 tx=0x7f3e2c009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:34.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.285+0000 7f3e2b7fe700 1 -- 172.21.15.6:0/956114880 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3e1c08c020 con 0x7f3e341003a0 2024-02-09T21:56:34.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.485+0000 7f3e3930c700 1 -- 172.21.15.6:0/956114880 --> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e3410f980 con 0x7f3e200692a0 2024-02-09T21:56:34.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.487+0000 7f3e2b7fe700 1 -- 172.21.15.6:0/956114880 <== mgr.14216 v2:172.21.15.6:6800/2624137529 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f3e3410f980 con 0x7f3e200692a0 2024-02-09T21:56:34.490 INFO:teuthology.orchestra.run.smithi006.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-02-09T21:56:34.491 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5330000H400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-02-09T21:56:34.491 INFO:teuthology.orchestra.run.smithi006.stdout:smithi090 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400UK400BGN 372G No 69s ago Has a FileSystem, LVM detected 2024-02-09T21:56:34.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.489+0000 7f3e3930c700 1 -- 172.21.15.6:0/956114880 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f3e200692a0 msgr2=0x7f3e2006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:34.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.489+0000 7f3e3930c700 1 --2- 172.21.15.6:0/956114880 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f3e200692a0 0x7f3e2006b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c004f90 tx=0x7f3e2c009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:34.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.489+0000 7f3e3930c700 1 -- 172.21.15.6:0/956114880 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e341003a0 msgr2=0x7f3e3410de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:34.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.489+0000 7f3e3930c700 1 --2- 172.21.15.6:0/956114880 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e341003a0 0x7f3e3410de70 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c002a50 tx=0x7f3e1c0046b0 comp rx=0 tx=0).stop 2024-02-09T21:56:34.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.489+0000 7f3e3930c700 1 -- 172.21.15.6:0/956114880 shutdown_connections 2024-02-09T21:56:34.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.489+0000 7f3e3930c700 1 --2- 172.21.15.6:0/956114880 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f3e200692a0 0x7f3e2006b750 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:34.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.489+0000 7f3e3930c700 1 --2- 172.21.15.6:0/956114880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e340fda80 0x7f3e3410d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:34.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.489+0000 7f3e3930c700 1 --2- 172.21.15.6:0/956114880 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e341003a0 0x7f3e3410de70 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:34.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.489+0000 7f3e3930c700 1 -- 172.21.15.6:0/956114880 >> 172.21.15.6:0/956114880 conn(0x7f3e340f74e0 msgr2=0x7f3e340f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:34.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.489+0000 7f3e3930c700 1 -- 172.21.15.6:0/956114880 shutdown_connections 2024-02-09T21:56:34.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:34.489+0000 7f3e3930c700 1 -- 172.21.15.6:0/956114880 wait complete. 2024-02-09T21:56:35.097 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-09T21:56:35.107 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi006.front.sepia.ceph.com 2024-02-09T21:56:35.108 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-02-09T21:56:35.368 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:56:36.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:35 smithi006 ceph-mon[29258]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:56:36.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:35 smithi006 ceph-mon[29258]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:36.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:35 smithi090 ceph-mon[36996]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:56:36.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:35 smithi090 ceph-mon[36996]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:36.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.952+0000 7f688d0f1700 1 -- 172.21.15.6:0/930990161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6888071eb0 msgr2=0x7f6888070320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:36.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.952+0000 7f688d0f1700 1 --2- 172.21.15.6:0/930990161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6888071eb0 0x7f6888070320 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6878009ce0 tx=0x7f6878004d10 comp rx=0 tx=0).stop 2024-02-09T21:56:36.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.952+0000 7f688d0f1700 1 -- 172.21.15.6:0/930990161 shutdown_connections 2024-02-09T21:56:36.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.952+0000 7f688d0f1700 1 --2- 172.21.15.6:0/930990161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6888070860 0x7f6888070cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:36.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.952+0000 7f688d0f1700 1 --2- 172.21.15.6:0/930990161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6888071eb0 0x7f6888070320 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:36.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.952+0000 7f688d0f1700 1 -- 172.21.15.6:0/930990161 >> 172.21.15.6:0/930990161 conn(0x7f68880f8620 msgr2=0x7f68880faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:36.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.952+0000 7f688d0f1700 1 -- 172.21.15.6:0/930990161 shutdown_connections 2024-02-09T21:56:36.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.952+0000 7f688d0f1700 1 -- 172.21.15.6:0/930990161 wait complete. 2024-02-09T21:56:36.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.952+0000 7f688d0f1700 1 Processor -- start 2024-02-09T21:56:36.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.952+0000 7f688d0f1700 1 -- start start 2024-02-09T21:56:36.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.953+0000 7f688d0f1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6888070860 0x7f688810fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:36.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.953+0000 7f688d0f1700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6888071eb0 0x7f6888110290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:36.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.953+0000 7f688d0f1700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6888110890 con 0x7f6888071eb0 2024-02-09T21:56:36.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.953+0000 7f688d0f1700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6888111020 con 0x7f6888070860 2024-02-09T21:56:36.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.953+0000 7f688659c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6888071eb0 0x7f6888110290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:36.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.953+0000 7f688659c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6888071eb0 0x7f6888110290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53662/0 (socket says 172.21.15.6:53662) 2024-02-09T21:56:36.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.953+0000 7f688659c700 1 -- 172.21.15.6:0/2294255939 learned_addr learned my addr 172.21.15.6:0/2294255939 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:56:36.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.953+0000 7f6886d9d700 1 --2- 172.21.15.6:0/2294255939 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6888070860 0x7f688810fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:36.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.953+0000 7f688659c700 1 -- 172.21.15.6:0/2294255939 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6888070860 msgr2=0x7f688810fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:36.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.953+0000 7f688659c700 1 --2- 172.21.15.6:0/2294255939 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6888070860 0x7f688810fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:36.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.953+0000 7f688659c700 1 -- 172.21.15.6:0/2294255939 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6878009940 con 0x7f6888071eb0 2024-02-09T21:56:36.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.953+0000 7f688659c700 1 --2- 172.21.15.6:0/2294255939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6888071eb0 0x7f6888110290 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6870002a50 tx=0x7f68700046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:36.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.953+0000 7f687f7fe700 1 -- 172.21.15.6:0/2294255939 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6870007cc0 con 0x7f6888071eb0 2024-02-09T21:56:36.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.953+0000 7f687f7fe700 1 -- 172.21.15.6:0/2294255939 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f687000f040 con 0x7f6888071eb0 2024-02-09T21:56:36.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.953+0000 7f688d0f1700 1 -- 172.21.15.6:0/2294255939 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6888111300 con 0x7f6888071eb0 2024-02-09T21:56:36.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.953+0000 7f687f7fe700 1 -- 172.21.15.6:0/2294255939 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6870009640 con 0x7f6888071eb0 2024-02-09T21:56:36.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.953+0000 7f688d0f1700 1 -- 172.21.15.6:0/2294255939 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6888111840 con 0x7f6888071eb0 2024-02-09T21:56:36.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.955+0000 7f688d0f1700 1 -- 172.21.15.6:0/2294255939 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f688804bed0 con 0x7f6888071eb0 2024-02-09T21:56:36.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.955+0000 7f687f7fe700 1 -- 172.21.15.6:0/2294255939 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f68700097c0 con 0x7f6888071eb0 2024-02-09T21:56:36.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.955+0000 7f687f7fe700 1 --2- 172.21.15.6:0/2294255939 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f68740692a0 0x7f687406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:36.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.955+0000 7f687f7fe700 1 -- 172.21.15.6:0/2294255939 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6870086ba0 con 0x7f6888071eb0 2024-02-09T21:56:36.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.955+0000 7f6886d9d700 1 --2- 172.21.15.6:0/2294255939 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f68740692a0 0x7f687406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:36.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.956+0000 7f6886d9d700 1 --2- 172.21.15.6:0/2294255939 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f68740692a0 0x7f687406b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f687800b740 tx=0x7f6878004d10 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:36.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:36.960+0000 7f687f7fe700 1 -- 172.21.15.6:0/2294255939 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6870056ca0 con 0x7f6888071eb0 2024-02-09T21:56:37.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:37.179+0000 7f688d0f1700 1 -- 172.21.15.6:0/2294255939 --> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f6888104550 con 0x7f68740692a0 2024-02-09T21:56:38.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:37 smithi006 ceph-mon[29258]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:38.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:37 smithi006 ceph-mon[29258]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:56:38.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:37 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-02-09T21:56:38.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:37 smithi090 ceph-mon[36996]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:38.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:37 smithi090 ceph-mon[36996]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:56:38.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:37 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-02-09T21:56:38.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:38.506+0000 7f687f7fe700 1 -- 172.21.15.6:0/2294255939 <== mgr.14216 v2:172.21.15.6:6800/2624137529 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6888104550 con 0x7f68740692a0 2024-02-09T21:56:38.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:38.510+0000 7f688d0f1700 1 -- 172.21.15.6:0/2294255939 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f68740692a0 msgr2=0x7f687406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:38.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:38.510+0000 7f688d0f1700 1 --2- 172.21.15.6:0/2294255939 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f68740692a0 0x7f687406b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f687800b740 tx=0x7f6878004d10 comp rx=0 tx=0).stop 2024-02-09T21:56:38.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:38.510+0000 7f688d0f1700 1 -- 172.21.15.6:0/2294255939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6888071eb0 msgr2=0x7f6888110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:38.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:38.510+0000 7f688d0f1700 1 --2- 172.21.15.6:0/2294255939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6888071eb0 0x7f6888110290 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6870002a50 tx=0x7f68700046b0 comp rx=0 tx=0).stop 2024-02-09T21:56:38.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:38.510+0000 7f688d0f1700 1 -- 172.21.15.6:0/2294255939 shutdown_connections 2024-02-09T21:56:38.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:38.510+0000 7f688d0f1700 1 --2- 172.21.15.6:0/2294255939 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f68740692a0 0x7f687406b750 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:38.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:38.510+0000 7f688d0f1700 1 --2- 172.21.15.6:0/2294255939 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6888070860 0x7f688810fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:38.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:38.510+0000 7f688d0f1700 1 --2- 172.21.15.6:0/2294255939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6888071eb0 0x7f6888110290 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:38.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:38.510+0000 7f688d0f1700 1 -- 172.21.15.6:0/2294255939 >> 172.21.15.6:0/2294255939 conn(0x7f68880f8620 msgr2=0x7f6888102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:38.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:38.511+0000 7f688d0f1700 1 -- 172.21.15.6:0/2294255939 shutdown_connections 2024-02-09T21:56:38.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:38.512+0000 7f688d0f1700 1 -- 172.21.15.6:0/2294255939 wait complete. 2024-02-09T21:56:38.880 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T21:56:38.485+0000 7fa7d4879700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-09T21:56:38.880 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:38 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-02-09T21:56:38.880 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:38 smithi006 ceph-mon[29258]: osdmap e41: 6 total, 6 up, 6 in 2024-02-09T21:56:38.880 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:38 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-02-09T21:56:38.880 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:38 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-02-09T21:56:38.880 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:38 smithi006 ceph-mon[29258]: osdmap e42: 6 total, 6 up, 6 in 2024-02-09T21:56:38.880 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:38 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-02-09T21:56:38.880 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:38 smithi006 ceph-mon[29258]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-09T21:56:38.880 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:38 smithi006 ceph-mon[29258]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-09T21:56:38.880 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:38 smithi006 ceph-mon[29258]: osdmap e43: 6 total, 6 up, 6 in 2024-02-09T21:56:38.881 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:38 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-02-09T21:56:38.881 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:38 smithi006 ceph-mon[29258]: fsmap cephfs:0 2024-02-09T21:56:38.881 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:38 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:38.881 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:38 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:56:39.187 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- bash -c 'ceph fs dump' 2024-02-09T21:56:39.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:38 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-02-09T21:56:39.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:38 smithi090 ceph-mon[36996]: osdmap e41: 6 total, 6 up, 6 in 2024-02-09T21:56:39.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:38 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-02-09T21:56:39.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:38 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-02-09T21:56:39.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:38 smithi090 ceph-mon[36996]: osdmap e42: 6 total, 6 up, 6 in 2024-02-09T21:56:39.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:38 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-02-09T21:56:39.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:38 smithi090 ceph-mon[36996]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-09T21:56:39.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:38 smithi090 ceph-mon[36996]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-09T21:56:39.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:38 smithi090 ceph-mon[36996]: osdmap e43: 6 total, 6 up, 6 in 2024-02-09T21:56:39.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:38 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-02-09T21:56:39.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:38 smithi090 ceph-mon[36996]: fsmap cephfs:0 2024-02-09T21:56:39.206 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:38 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:39.206 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:38 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:56:40.111 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:39 smithi006 ceph-mon[29258]: Saving service mds.cephfs spec with placement count:4 2024-02-09T21:56:40.111 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:39 smithi006 ceph-mon[29258]: pgmap v172: 65 pgs: 11 creating+peering, 1 active+clean, 53 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:40.111 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:39 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:56:40.111 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:39 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:56:40.111 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:39 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:40.111 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:39 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi006.ypgiqs", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T21:56:40.112 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:39 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi006.ypgiqs", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T21:56:40.112 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:39 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:56:40.112 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:39 smithi006 ceph-mon[29258]: Deploying daemon mds.cephfs.smithi006.ypgiqs on smithi006 2024-02-09T21:56:40.112 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:39 smithi006 ceph-mon[29258]: osdmap e44: 6 total, 6 up, 6 in 2024-02-09T21:56:40.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:39 smithi090 ceph-mon[36996]: Saving service mds.cephfs spec with placement count:4 2024-02-09T21:56:40.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:39 smithi090 ceph-mon[36996]: pgmap v172: 65 pgs: 11 creating+peering, 1 active+clean, 53 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:40.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:39 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:56:40.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:39 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:56:40.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:39 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:40.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:39 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi006.ypgiqs", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T21:56:40.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:39 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi006.ypgiqs", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T21:56:40.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:39 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:56:40.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:39 smithi090 ceph-mon[36996]: Deploying daemon mds.cephfs.smithi006.ypgiqs on smithi006 2024-02-09T21:56:40.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:39 smithi090 ceph-mon[36996]: osdmap e44: 6 total, 6 up, 6 in 2024-02-09T21:56:40.511 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:56:41.704 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:41 smithi090 ceph-mon[36996]: osdmap e45: 6 total, 6 up, 6 in 2024-02-09T21:56:41.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:41 smithi090 ceph-mon[36996]: pgmap v175: 65 pgs: 15 creating+peering, 6 active+clean, 44 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:41.719 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:41 smithi006 ceph-mon[29258]: osdmap e45: 6 total, 6 up, 6 in 2024-02-09T21:56:41.720 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:41 smithi006 ceph-mon[29258]: pgmap v175: 65 pgs: 15 creating+peering, 6 active+clean, 44 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:42.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.767+0000 7f10ae662700 1 -- 172.21.15.6:0/2673030104 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10a80fbe40 msgr2=0x7f10a80fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:42.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.767+0000 7f10ae662700 1 --2- 172.21.15.6:0/2673030104 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10a80fbe40 0x7f10a80fc250 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1098004940 tx=0x7f1098009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:42.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.767+0000 7f10ae662700 1 -- 172.21.15.6:0/2673030104 shutdown_connections 2024-02-09T21:56:42.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.767+0000 7f10ae662700 1 --2- 172.21.15.6:0/2673030104 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10a80fcfb0 0x7f10a80fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:42.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.767+0000 7f10ae662700 1 --2- 172.21.15.6:0/2673030104 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10a80fbe40 0x7f10a80fc250 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:42.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.767+0000 7f10ae662700 1 -- 172.21.15.6:0/2673030104 >> 172.21.15.6:0/2673030104 conn(0x7f10a80f74e0 msgr2=0x7f10a80f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:42.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.767+0000 7f10ae662700 1 -- 172.21.15.6:0/2673030104 shutdown_connections 2024-02-09T21:56:42.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.767+0000 7f10ae662700 1 -- 172.21.15.6:0/2673030104 wait complete. 2024-02-09T21:56:42.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.768+0000 7f10ae662700 1 Processor -- start 2024-02-09T21:56:42.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.768+0000 7f10ae662700 1 -- start start 2024-02-09T21:56:42.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.768+0000 7f10ae662700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10a80fbe40 0x7f10a810c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:42.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.768+0000 7f10ae662700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10a80fcfb0 0x7f10a810cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:42.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.768+0000 7f10ae662700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a810d310 con 0x7f10a80fbe40 2024-02-09T21:56:42.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.768+0000 7f10ae662700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a810dbb0 con 0x7f10a80fcfb0 2024-02-09T21:56:42.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.768+0000 7f10a7fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10a80fbe40 0x7f10a810c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:42.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.768+0000 7f10a7fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10a80fbe40 0x7f10a810c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34900/0 (socket says 172.21.15.6:34900) 2024-02-09T21:56:42.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.768+0000 7f10a7fff700 1 -- 172.21.15.6:0/840672566 learned_addr learned my addr 172.21.15.6:0/840672566 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:56:42.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.768+0000 7f10a77fe700 1 --2- 172.21.15.6:0/840672566 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10a80fcfb0 0x7f10a810cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:42.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.769+0000 7f10a7fff700 1 -- 172.21.15.6:0/840672566 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10a80fcfb0 msgr2=0x7f10a810cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:42.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.769+0000 7f10a7fff700 1 --2- 172.21.15.6:0/840672566 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10a80fcfb0 0x7f10a810cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:42.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.769+0000 7f10a7fff700 1 -- 172.21.15.6:0/840672566 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1098005040 con 0x7f10a80fbe40 2024-02-09T21:56:42.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.769+0000 7f10a7fff700 1 --2- 172.21.15.6:0/840672566 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10a80fbe40 0x7f10a810c7d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1098000c00 tx=0x7f1098009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:42.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.769+0000 7f10a4ff9700 1 -- 172.21.15.6:0/840672566 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1098021070 con 0x7f10a80fbe40 2024-02-09T21:56:42.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.769+0000 7f10a4ff9700 1 -- 172.21.15.6:0/840672566 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f109801c400 con 0x7f10a80fbe40 2024-02-09T21:56:42.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.769+0000 7f10a4ff9700 1 -- 172.21.15.6:0/840672566 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f109800e070 con 0x7f10a80fbe40 2024-02-09T21:56:42.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.769+0000 7f10ae662700 1 -- 172.21.15.6:0/840672566 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10a810de30 con 0x7f10a80fbe40 2024-02-09T21:56:42.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.769+0000 7f10ae662700 1 -- 172.21.15.6:0/840672566 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10a810e310 con 0x7f10a80fbe40 2024-02-09T21:56:42.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.770+0000 7f10ae662700 1 -- 172.21.15.6:0/840672566 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10a804bed0 con 0x7f10a80fbe40 2024-02-09T21:56:42.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.770+0000 7f10a4ff9700 1 -- 172.21.15.6:0/840672566 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f109801ca50 con 0x7f10a80fbe40 2024-02-09T21:56:42.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.771+0000 7f10a4ff9700 1 --2- 172.21.15.6:0/840672566 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f10900692a0 0x7f109006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:42.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.771+0000 7f10a4ff9700 1 -- 172.21.15.6:0/840672566 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f10980882b0 con 0x7f10a80fbe40 2024-02-09T21:56:42.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.771+0000 7f10a77fe700 1 --2- 172.21.15.6:0/840672566 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f10900692a0 0x7f109006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:42.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.771+0000 7f10a77fe700 1 --2- 172.21.15.6:0/840672566 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f10900692a0 0x7f109006b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f109c009b80 tx=0x7f109c00b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:42.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:42.776+0000 7f10a4ff9700 1 -- 172.21.15.6:0/840672566 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1098058110 con 0x7f10a80fbe40 2024-02-09T21:56:43.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:43.017+0000 7f10ae662700 1 -- 172.21.15.6:0/840672566 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f10a8104930 con 0x7f10a80fbe40 2024-02-09T21:56:43.021 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:42 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:43.022 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:42 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:56:43.022 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:42 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:43.022 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:42 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:43.022 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:42 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:43.023 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:42 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi090.cgsuda", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T21:56:43.023 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:42 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi090.cgsuda", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T21:56:43.023 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:42 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:56:43.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:43.017+0000 7f10a4ff9700 1 -- 172.21.15.6:0/840672566 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 4 v4) v1 ==== 75+0+1243 (secure 0 0 0) 0x7f1098057ca0 con 0x7f10a80fbe40 2024-02-09T21:56:43.024 INFO:teuthology.orchestra.run.smithi006.stdout:e4 2024-02-09T21:56:43.024 INFO:teuthology.orchestra.run.smithi006.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-09T21:56:43.025 INFO:teuthology.orchestra.run.smithi006.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-09T21:56:43.025 INFO:teuthology.orchestra.run.smithi006.stdout:legacy client fscid: 1 2024-02-09T21:56:43.025 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:56:43.025 INFO:teuthology.orchestra.run.smithi006.stdout:Filesystem 'cephfs' (1) 2024-02-09T21:56:43.025 INFO:teuthology.orchestra.run.smithi006.stdout:fs_name cephfs 2024-02-09T21:56:43.025 INFO:teuthology.orchestra.run.smithi006.stdout:epoch 4 2024-02-09T21:56:43.025 INFO:teuthology.orchestra.run.smithi006.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-09T21:56:43.026 INFO:teuthology.orchestra.run.smithi006.stdout:created 2024-02-09T21:56:38.486224+0000 2024-02-09T21:56:43.026 INFO:teuthology.orchestra.run.smithi006.stdout:modified 2024-02-09T21:56:42.885227+0000 2024-02-09T21:56:43.026 INFO:teuthology.orchestra.run.smithi006.stdout:tableserver 0 2024-02-09T21:56:43.026 INFO:teuthology.orchestra.run.smithi006.stdout:root 0 2024-02-09T21:56:43.026 INFO:teuthology.orchestra.run.smithi006.stdout:session_timeout 60 2024-02-09T21:56:43.026 INFO:teuthology.orchestra.run.smithi006.stdout:session_autoclose 300 2024-02-09T21:56:43.027 INFO:teuthology.orchestra.run.smithi006.stdout:max_file_size 1099511627776 2024-02-09T21:56:43.027 INFO:teuthology.orchestra.run.smithi006.stdout:required_client_features {} 2024-02-09T21:56:43.027 INFO:teuthology.orchestra.run.smithi006.stdout:last_failure 0 2024-02-09T21:56:43.027 INFO:teuthology.orchestra.run.smithi006.stdout:last_failure_osd_epoch 0 2024-02-09T21:56:43.027 INFO:teuthology.orchestra.run.smithi006.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-09T21:56:43.027 INFO:teuthology.orchestra.run.smithi006.stdout:max_mds 1 2024-02-09T21:56:43.027 INFO:teuthology.orchestra.run.smithi006.stdout:in 0 2024-02-09T21:56:43.028 INFO:teuthology.orchestra.run.smithi006.stdout:up {0=14458} 2024-02-09T21:56:43.028 INFO:teuthology.orchestra.run.smithi006.stdout:failed 2024-02-09T21:56:43.028 INFO:teuthology.orchestra.run.smithi006.stdout:damaged 2024-02-09T21:56:43.028 INFO:teuthology.orchestra.run.smithi006.stdout:stopped 2024-02-09T21:56:43.028 INFO:teuthology.orchestra.run.smithi006.stdout:data_pools [3] 2024-02-09T21:56:43.028 INFO:teuthology.orchestra.run.smithi006.stdout:metadata_pool 2 2024-02-09T21:56:43.029 INFO:teuthology.orchestra.run.smithi006.stdout:inline_data disabled 2024-02-09T21:56:43.029 INFO:teuthology.orchestra.run.smithi006.stdout:balancer 2024-02-09T21:56:43.029 INFO:teuthology.orchestra.run.smithi006.stdout:standby_count_wanted 0 2024-02-09T21:56:43.029 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.cephfs.smithi006.ypgiqs{0:14458} state up:creating seq 1 addr [v2:172.21.15.6:6826/669315024,v1:172.21.15.6:6827/669315024] compat {c=[1],r=[1],i=[7ff]}] 2024-02-09T21:56:43.029 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:56:43.029 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:56:43.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:43.019+0000 7f10ae662700 1 -- 172.21.15.6:0/840672566 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f10900692a0 msgr2=0x7f109006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:43.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:43.019+0000 7f10ae662700 1 --2- 172.21.15.6:0/840672566 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f10900692a0 0x7f109006b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f109c009b80 tx=0x7f109c00b040 comp rx=0 tx=0).stop 2024-02-09T21:56:43.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:43.019+0000 7f10ae662700 1 -- 172.21.15.6:0/840672566 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10a80fbe40 msgr2=0x7f10a810c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:43.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:43.019+0000 7f10ae662700 1 --2- 172.21.15.6:0/840672566 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10a80fbe40 0x7f10a810c7d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1098000c00 tx=0x7f1098009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:43.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:43.019+0000 7f10ae662700 1 -- 172.21.15.6:0/840672566 shutdown_connections 2024-02-09T21:56:43.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:43.019+0000 7f10ae662700 1 --2- 172.21.15.6:0/840672566 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f10900692a0 0x7f109006b750 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:43.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:43.019+0000 7f10ae662700 1 --2- 172.21.15.6:0/840672566 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10a80fbe40 0x7f10a810c7d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:43.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:43.019+0000 7f10ae662700 1 --2- 172.21.15.6:0/840672566 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10a80fcfb0 0x7f10a810cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:43.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:43.019+0000 7f10ae662700 1 -- 172.21.15.6:0/840672566 >> 172.21.15.6:0/840672566 conn(0x7f10a80f74e0 msgr2=0x7f10a8100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:43.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:43.019+0000 7f10ae662700 1 -- 172.21.15.6:0/840672566 shutdown_connections 2024-02-09T21:56:43.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:43.019+0000 7f10ae662700 1 -- 172.21.15.6:0/840672566 wait complete. 2024-02-09T21:56:43.033 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 4 2024-02-09T21:56:43.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:42 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:43.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:42 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:56:43.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:42 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:43.206 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:42 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:43.206 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:42 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:43.206 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:42 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi090.cgsuda", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T21:56:43.206 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:42 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi090.cgsuda", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T21:56:43.206 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:42 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:56:43.690 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-09T21:56:43.700 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi006.front.sepia.ceph.com 2024-02-09T21:56:43.701 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- bash -c 'ceph fs set cephfs max_mds 2' 2024-02-09T21:56:43.968 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:56:44.082 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:43 smithi006 ceph-mon[29258]: Deploying daemon mds.cephfs.smithi090.cgsuda on smithi090 2024-02-09T21:56:44.082 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:43 smithi006 ceph-mon[29258]: pgmap v176: 65 pgs: 15 creating+peering, 33 active+clean, 17 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:44.082 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:43 smithi006 ceph-mon[29258]: mds.? [v2:172.21.15.6:6826/669315024,v1:172.21.15.6:6827/669315024] up:boot 2024-02-09T21:56:44.082 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:43 smithi006 ceph-mon[29258]: daemon mds.cephfs.smithi006.ypgiqs assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-02-09T21:56:44.082 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:43 smithi006 ceph-mon[29258]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-09T21:56:44.082 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:43 smithi006 ceph-mon[29258]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-09T21:56:44.082 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:43 smithi006 ceph-mon[29258]: Cluster is now healthy 2024-02-09T21:56:44.082 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:43 smithi006 ceph-mon[29258]: fsmap cephfs:0 1 up:standby 2024-02-09T21:56:44.082 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:43 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi006.ypgiqs"}]: dispatch 2024-02-09T21:56:44.082 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:43 smithi006 ceph-mon[29258]: fsmap cephfs:1 {0=cephfs.smithi006.ypgiqs=up:creating} 2024-02-09T21:56:44.082 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:43 smithi006 ceph-mon[29258]: daemon mds.cephfs.smithi006.ypgiqs is now active in filesystem cephfs as rank 0 2024-02-09T21:56:44.083 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:43 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/840672566' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-09T21:56:44.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:43 smithi090 ceph-mon[36996]: Deploying daemon mds.cephfs.smithi090.cgsuda on smithi090 2024-02-09T21:56:44.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:43 smithi090 ceph-mon[36996]: pgmap v176: 65 pgs: 15 creating+peering, 33 active+clean, 17 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-09T21:56:44.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:43 smithi090 ceph-mon[36996]: mds.? [v2:172.21.15.6:6826/669315024,v1:172.21.15.6:6827/669315024] up:boot 2024-02-09T21:56:44.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:43 smithi090 ceph-mon[36996]: daemon mds.cephfs.smithi006.ypgiqs assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-02-09T21:56:44.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:43 smithi090 ceph-mon[36996]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-09T21:56:44.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:43 smithi090 ceph-mon[36996]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-09T21:56:44.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:43 smithi090 ceph-mon[36996]: Cluster is now healthy 2024-02-09T21:56:44.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:43 smithi090 ceph-mon[36996]: fsmap cephfs:0 1 up:standby 2024-02-09T21:56:44.206 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:43 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi006.ypgiqs"}]: dispatch 2024-02-09T21:56:44.206 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:43 smithi090 ceph-mon[36996]: fsmap cephfs:1 {0=cephfs.smithi006.ypgiqs=up:creating} 2024-02-09T21:56:44.206 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:43 smithi090 ceph-mon[36996]: daemon mds.cephfs.smithi006.ypgiqs is now active in filesystem cephfs as rank 0 2024-02-09T21:56:44.206 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:43 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/840672566' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-09T21:56:45.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:44 smithi006 ceph-mon[29258]: mds.? [v2:172.21.15.6:6826/669315024,v1:172.21.15.6:6827/669315024] up:active 2024-02-09T21:56:45.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:44 smithi006 ceph-mon[29258]: fsmap cephfs:1 {0=cephfs.smithi006.ypgiqs=up:active} 2024-02-09T21:56:45.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:44 smithi090 ceph-mon[36996]: mds.? [v2:172.21.15.6:6826/669315024,v1:172.21.15.6:6827/669315024] up:active 2024-02-09T21:56:45.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:44 smithi090 ceph-mon[36996]: fsmap cephfs:1 {0=cephfs.smithi006.ypgiqs=up:active} 2024-02-09T21:56:45.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.532+0000 7ff02f6ac700 1 -- 172.21.15.6:0/2909020417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0280fcfb0 msgr2=0x7ff0280fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:45.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.532+0000 7ff02f6ac700 1 --2- 172.21.15.6:0/2909020417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0280fcfb0 0x7ff0280fd420 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff018004940 tx=0x7ff018009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:45.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.532+0000 7ff02f6ac700 1 -- 172.21.15.6:0/2909020417 shutdown_connections 2024-02-09T21:56:45.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.532+0000 7ff02f6ac700 1 --2- 172.21.15.6:0/2909020417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0280fcfb0 0x7ff0280fd420 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:45.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.532+0000 7ff02f6ac700 1 --2- 172.21.15.6:0/2909020417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0280fbe40 0x7ff0280fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:45.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.532+0000 7ff02f6ac700 1 -- 172.21.15.6:0/2909020417 >> 172.21.15.6:0/2909020417 conn(0x7ff0280f74e0 msgr2=0x7ff0280f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:45.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.533+0000 7ff02f6ac700 1 -- 172.21.15.6:0/2909020417 shutdown_connections 2024-02-09T21:56:45.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.533+0000 7ff02f6ac700 1 -- 172.21.15.6:0/2909020417 wait complete. 2024-02-09T21:56:45.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.533+0000 7ff02f6ac700 1 Processor -- start 2024-02-09T21:56:45.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.533+0000 7ff02f6ac700 1 -- start start 2024-02-09T21:56:45.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.533+0000 7ff02f6ac700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0280fbe40 0x7ff02810d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:45.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.533+0000 7ff02f6ac700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0280fcfb0 0x7ff02810de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:45.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.533+0000 7ff02f6ac700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff02810e400 con 0x7ff0280fcfb0 2024-02-09T21:56:45.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.533+0000 7ff02f6ac700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff02810eca0 con 0x7ff0280fbe40 2024-02-09T21:56:45.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.533+0000 7ff02cc47700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0280fcfb0 0x7ff02810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:45.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.533+0000 7ff02cc47700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0280fcfb0 0x7ff02810de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34924/0 (socket says 172.21.15.6:34924) 2024-02-09T21:56:45.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.534+0000 7ff02cc47700 1 -- 172.21.15.6:0/316874894 learned_addr learned my addr 172.21.15.6:0/316874894 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:56:45.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.534+0000 7ff02d448700 1 --2- 172.21.15.6:0/316874894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0280fbe40 0x7ff02810d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:45.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.534+0000 7ff02d448700 1 -- 172.21.15.6:0/316874894 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0280fcfb0 msgr2=0x7ff02810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:45.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.534+0000 7ff02d448700 1 --2- 172.21.15.6:0/316874894 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0280fcfb0 0x7ff02810de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:45.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.534+0000 7ff02d448700 1 -- 172.21.15.6:0/316874894 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff018005040 con 0x7ff0280fbe40 2024-02-09T21:56:45.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.534+0000 7ff02d448700 1 --2- 172.21.15.6:0/316874894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0280fbe40 0x7ff02810d8c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff02400ec20 tx=0x7ff02400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:45.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.534+0000 7ff01dffb700 1 -- 172.21.15.6:0/316874894 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff024007c00 con 0x7ff0280fbe40 2024-02-09T21:56:45.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.534+0000 7ff01dffb700 1 -- 172.21.15.6:0/316874894 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff02400f040 con 0x7ff0280fbe40 2024-02-09T21:56:45.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.534+0000 7ff02f6ac700 1 -- 172.21.15.6:0/316874894 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff02810ef80 con 0x7ff0280fbe40 2024-02-09T21:56:45.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.534+0000 7ff01dffb700 1 -- 172.21.15.6:0/316874894 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff02400cc30 con 0x7ff0280fbe40 2024-02-09T21:56:45.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.534+0000 7ff02f6ac700 1 -- 172.21.15.6:0/316874894 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff02810f4c0 con 0x7ff0280fbe40 2024-02-09T21:56:45.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.536+0000 7ff02f6ac700 1 -- 172.21.15.6:0/316874894 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff028107980 con 0x7ff0280fbe40 2024-02-09T21:56:45.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.536+0000 7ff01dffb700 1 -- 172.21.15.6:0/316874894 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7ff024014480 con 0x7ff0280fbe40 2024-02-09T21:56:45.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.536+0000 7ff01dffb700 1 --2- 172.21.15.6:0/316874894 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7ff0140692a0 0x7ff01406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:45.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.536+0000 7ff01dffb700 1 -- 172.21.15.6:0/316874894 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7ff0240875c0 con 0x7ff0280fbe40 2024-02-09T21:56:45.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.536+0000 7ff02cc47700 1 --2- 172.21.15.6:0/316874894 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7ff0140692a0 0x7ff01406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:45.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.537+0000 7ff02cc47700 1 --2- 172.21.15.6:0/316874894 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7ff0140692a0 0x7ff01406b750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff01800b0d0 tx=0x7ff018009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:45.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.541+0000 7ff01dffb700 1 -- 172.21.15.6:0/316874894 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff024057430 con 0x7ff0280fbe40 2024-02-09T21:56:45.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.789+0000 7ff02f6ac700 1 -- 172.21.15.6:0/316874894 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7ff02810f730 con 0x7ff0280fbe40 2024-02-09T21:56:45.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.991+0000 7ff01dffb700 1 -- 172.21.15.6:0/316874894 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v8) v1 ==== 105+0+0 (secure 0 0 0) 0x7ff024056fc0 con 0x7ff0280fbe40 2024-02-09T21:56:45.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.993+0000 7ff02f6ac700 1 -- 172.21.15.6:0/316874894 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7ff0140692a0 msgr2=0x7ff01406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:45.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.993+0000 7ff02f6ac700 1 --2- 172.21.15.6:0/316874894 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7ff0140692a0 0x7ff01406b750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff01800b0d0 tx=0x7ff018009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:45.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.993+0000 7ff02f6ac700 1 -- 172.21.15.6:0/316874894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0280fbe40 msgr2=0x7ff02810d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:45.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.993+0000 7ff02f6ac700 1 --2- 172.21.15.6:0/316874894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0280fbe40 0x7ff02810d8c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff02400ec20 tx=0x7ff02400b040 comp rx=0 tx=0).stop 2024-02-09T21:56:45.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.993+0000 7ff02f6ac700 1 -- 172.21.15.6:0/316874894 shutdown_connections 2024-02-09T21:56:45.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.993+0000 7ff02f6ac700 1 --2- 172.21.15.6:0/316874894 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7ff0140692a0 0x7ff01406b750 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:45.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.993+0000 7ff02f6ac700 1 --2- 172.21.15.6:0/316874894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0280fbe40 0x7ff02810d8c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:45.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.993+0000 7ff02f6ac700 1 --2- 172.21.15.6:0/316874894 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0280fcfb0 0x7ff02810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:45.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.994+0000 7ff02f6ac700 1 -- 172.21.15.6:0/316874894 >> 172.21.15.6:0/316874894 conn(0x7ff0280f74e0 msgr2=0x7ff0280f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:45.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.994+0000 7ff02f6ac700 1 -- 172.21.15.6:0/316874894 shutdown_connections 2024-02-09T21:56:45.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:45.994+0000 7ff02f6ac700 1 -- 172.21.15.6:0/316874894 wait complete. 2024-02-09T21:56:46.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:45 smithi006 ceph-mon[29258]: pgmap v177: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-02-09T21:56:46.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:45 smithi006 ceph-mon[29258]: mds.? [v2:172.21.15.90:6824/2739488453,v1:172.21.15.90:6825/2739488453] up:boot 2024-02-09T21:56:46.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:45 smithi006 ceph-mon[29258]: fsmap cephfs:1 {0=cephfs.smithi006.ypgiqs=up:active} 1 up:standby 2024-02-09T21:56:46.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:45 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi090.cgsuda"}]: dispatch 2024-02-09T21:56:46.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:45 smithi006 ceph-mon[29258]: fsmap cephfs:1 {0=cephfs.smithi006.ypgiqs=up:active} 1 up:standby 2024-02-09T21:56:46.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:45 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:46.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:45 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:46.159 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:45 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:46.159 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:45 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi006.qyamnu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T21:56:46.159 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:45 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi006.qyamnu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T21:56:46.159 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:45 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:56:46.159 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:45 smithi006 ceph-mon[29258]: Deploying daemon mds.cephfs.smithi006.qyamnu on smithi006 2024-02-09T21:56:46.159 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:45 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/316874894' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-02-09T21:56:46.160 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:45 smithi006 ceph-mon[29258]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-02-09T21:56:46.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:45 smithi090 ceph-mon[36996]: pgmap v177: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-02-09T21:56:46.207 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:45 smithi090 ceph-mon[36996]: mds.? [v2:172.21.15.90:6824/2739488453,v1:172.21.15.90:6825/2739488453] up:boot 2024-02-09T21:56:46.207 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:45 smithi090 ceph-mon[36996]: fsmap cephfs:1 {0=cephfs.smithi006.ypgiqs=up:active} 1 up:standby 2024-02-09T21:56:46.207 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:45 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi090.cgsuda"}]: dispatch 2024-02-09T21:56:46.207 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:45 smithi090 ceph-mon[36996]: fsmap cephfs:1 {0=cephfs.smithi006.ypgiqs=up:active} 1 up:standby 2024-02-09T21:56:46.207 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:45 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:46.207 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:45 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:46.207 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:45 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:46.207 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:45 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi006.qyamnu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T21:56:46.207 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:45 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi006.qyamnu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T21:56:46.208 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:45 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:56:46.208 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:45 smithi090 ceph-mon[36996]: Deploying daemon mds.cephfs.smithi006.qyamnu on smithi006 2024-02-09T21:56:46.208 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:45 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/316874894' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-02-09T21:56:46.208 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:45 smithi090 ceph-mon[36996]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-02-09T21:56:47.154 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:46 smithi006 ceph-mon[29258]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-09T21:56:47.155 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:46 smithi006 ceph-mon[29258]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-02-09T21:56:47.155 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:46 smithi006 ceph-mon[29258]: daemon mds.cephfs.smithi090.cgsuda assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-02-09T21:56:47.155 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:46 smithi006 ceph-mon[29258]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-02-09T21:56:47.155 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:46 smithi006 ceph-mon[29258]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-09T21:56:47.155 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:46 smithi006 ceph-mon[29258]: fsmap cephfs:1 {0=cephfs.smithi006.ypgiqs=up:active} 1 up:standby 2024-02-09T21:56:47.155 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:46 smithi006 ceph-mon[29258]: fsmap cephfs:2 {0=cephfs.smithi006.ypgiqs=up:active,1=cephfs.smithi090.cgsuda=up:creating} 2024-02-09T21:56:47.155 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:46 smithi006 ceph-mon[29258]: daemon mds.cephfs.smithi090.cgsuda is now active in filesystem cephfs as rank 1 2024-02-09T21:56:47.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:46 smithi090 ceph-mon[36996]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-09T21:56:47.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:46 smithi090 ceph-mon[36996]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-02-09T21:56:47.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:46 smithi090 ceph-mon[36996]: daemon mds.cephfs.smithi090.cgsuda assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-02-09T21:56:47.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:46 smithi090 ceph-mon[36996]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-02-09T21:56:47.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:46 smithi090 ceph-mon[36996]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-09T21:56:47.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:46 smithi090 ceph-mon[36996]: fsmap cephfs:1 {0=cephfs.smithi006.ypgiqs=up:active} 1 up:standby 2024-02-09T21:56:47.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:46 smithi090 ceph-mon[36996]: fsmap cephfs:2 {0=cephfs.smithi006.ypgiqs=up:active,1=cephfs.smithi090.cgsuda=up:creating} 2024-02-09T21:56:47.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:46 smithi090 ceph-mon[36996]: daemon mds.cephfs.smithi090.cgsuda is now active in filesystem cephfs as rank 1 2024-02-09T21:56:47.640 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-09T21:56:47.653 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi006.front.sepia.ceph.com 2024-02-09T21:56:47.653 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- bash -c 'ceph fs set cephfs allow_standby_replay true' 2024-02-09T21:56:48.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:48 smithi006 ceph-mon[29258]: pgmap v178: 65 pgs: 65 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s wr, 4 op/s 2024-02-09T21:56:48.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:48 smithi006 ceph-mon[29258]: mds.? [v2:172.21.15.90:6824/2739488453,v1:172.21.15.90:6825/2739488453] up:active 2024-02-09T21:56:48.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:48 smithi006 ceph-mon[29258]: fsmap cephfs:2 {0=cephfs.smithi006.ypgiqs=up:active,1=cephfs.smithi090.cgsuda=up:active} 2024-02-09T21:56:48.262 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:56:48.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:48 smithi090 ceph-mon[36996]: pgmap v178: 65 pgs: 65 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s wr, 4 op/s 2024-02-09T21:56:48.456 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:48 smithi090 ceph-mon[36996]: mds.? [v2:172.21.15.90:6824/2739488453,v1:172.21.15.90:6825/2739488453] up:active 2024-02-09T21:56:48.456 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:48 smithi090 ceph-mon[36996]: fsmap cephfs:2 {0=cephfs.smithi006.ypgiqs=up:active,1=cephfs.smithi090.cgsuda=up:active} 2024-02-09T21:56:49.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:49 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:49.456 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:49 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:49.456 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:49 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:49.456 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:49 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi090.mipeaa", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T21:56:49.456 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:49 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi090.mipeaa", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T21:56:49.456 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:49 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:56:49.456 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:49 smithi090 ceph-mon[36996]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-02-09T21:56:49.457 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:49 smithi090 ceph-mon[36996]: Cluster is now healthy 2024-02-09T21:56:49.457 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:49 smithi090 ceph-mon[36996]: mds.? [v2:172.21.15.6:6828/3025444869,v1:172.21.15.6:6829/3025444869] up:boot 2024-02-09T21:56:49.457 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:49 smithi090 ceph-mon[36996]: fsmap cephfs:2 {0=cephfs.smithi006.ypgiqs=up:active,1=cephfs.smithi090.cgsuda=up:active} 1 up:standby 2024-02-09T21:56:49.457 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:49 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi006.qyamnu"}]: dispatch 2024-02-09T21:56:49.561 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:49 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:49.561 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:49 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:49.561 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:49 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:49.561 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:49 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi090.mipeaa", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T21:56:49.562 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:49 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi090.mipeaa", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T21:56:49.562 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:49 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:56:49.562 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:49 smithi006 ceph-mon[29258]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-02-09T21:56:49.562 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:49 smithi006 ceph-mon[29258]: Cluster is now healthy 2024-02-09T21:56:49.562 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:49 smithi006 ceph-mon[29258]: mds.? [v2:172.21.15.6:6828/3025444869,v1:172.21.15.6:6829/3025444869] up:boot 2024-02-09T21:56:49.562 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:49 smithi006 ceph-mon[29258]: fsmap cephfs:2 {0=cephfs.smithi006.ypgiqs=up:active,1=cephfs.smithi090.cgsuda=up:active} 1 up:standby 2024-02-09T21:56:49.562 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:49 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi006.qyamnu"}]: dispatch 2024-02-09T21:56:49.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.761+0000 7fc181f47700 1 -- 172.21.15.6:0/1752801246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc17c0fe0e0 msgr2=0x7fc17c0fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:49.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.761+0000 7fc181f47700 1 --2- 172.21.15.6:0/1752801246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc17c0fe0e0 0x7fc17c0fe550 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc16c004940 tx=0x7fc16c009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:49.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.761+0000 7fc181f47700 1 -- 172.21.15.6:0/1752801246 shutdown_connections 2024-02-09T21:56:49.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.761+0000 7fc181f47700 1 --2- 172.21.15.6:0/1752801246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc17c0fe0e0 0x7fc17c0fe550 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:49.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.761+0000 7fc181f47700 1 --2- 172.21.15.6:0/1752801246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc17c0fcef0 0x7fc17c0fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:49.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.761+0000 7fc181f47700 1 -- 172.21.15.6:0/1752801246 >> 172.21.15.6:0/1752801246 conn(0x7fc17c0f8620 msgr2=0x7fc17c0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:49.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.761+0000 7fc181f47700 1 -- 172.21.15.6:0/1752801246 shutdown_connections 2024-02-09T21:56:49.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.761+0000 7fc181f47700 1 -- 172.21.15.6:0/1752801246 wait complete. 2024-02-09T21:56:49.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.761+0000 7fc181f47700 1 Processor -- start 2024-02-09T21:56:49.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.762+0000 7fc181f47700 1 -- start start 2024-02-09T21:56:49.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.762+0000 7fc181f47700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc17c0fcef0 0x7fc17c10d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:49.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.762+0000 7fc181f47700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc17c0fe0e0 0x7fc17c10deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:49.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.762+0000 7fc181f47700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c10e4b0 con 0x7fc17c0fcef0 2024-02-09T21:56:49.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.762+0000 7fc181f47700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c10ed50 con 0x7fc17c0fe0e0 2024-02-09T21:56:49.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.762+0000 7fc17b7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc17c0fcef0 0x7fc17c10d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:49.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.762+0000 7fc17b7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc17c0fcef0 0x7fc17c10d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44158/0 (socket says 172.21.15.6:44158) 2024-02-09T21:56:49.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.762+0000 7fc17b7fe700 1 -- 172.21.15.6:0/3799162477 learned_addr learned my addr 172.21.15.6:0/3799162477 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:56:49.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.762+0000 7fc17affd700 1 --2- 172.21.15.6:0/3799162477 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc17c0fe0e0 0x7fc17c10deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:49.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.762+0000 7fc17b7fe700 1 -- 172.21.15.6:0/3799162477 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc17c0fe0e0 msgr2=0x7fc17c10deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:49.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.762+0000 7fc17b7fe700 1 --2- 172.21.15.6:0/3799162477 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc17c0fe0e0 0x7fc17c10deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:49.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.762+0000 7fc17b7fe700 1 -- 172.21.15.6:0/3799162477 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc16c005040 con 0x7fc17c0fcef0 2024-02-09T21:56:49.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.762+0000 7fc17b7fe700 1 --2- 172.21.15.6:0/3799162477 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc17c0fcef0 0x7fc17c10d970 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc170002a80 tx=0x7fc170004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:49.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.763+0000 7fc163fff700 1 -- 172.21.15.6:0/3799162477 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc170007cc0 con 0x7fc17c0fcef0 2024-02-09T21:56:49.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.763+0000 7fc181f47700 1 -- 172.21.15.6:0/3799162477 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc17c10f030 con 0x7fc17c0fcef0 2024-02-09T21:56:49.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.763+0000 7fc163fff700 1 -- 172.21.15.6:0/3799162477 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc17000f040 con 0x7fc17c0fcef0 2024-02-09T21:56:49.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.763+0000 7fc163fff700 1 -- 172.21.15.6:0/3799162477 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc17000da30 con 0x7fc17c0fcef0 2024-02-09T21:56:49.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.763+0000 7fc181f47700 1 -- 172.21.15.6:0/3799162477 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc17c10f570 con 0x7fc17c0fcef0 2024-02-09T21:56:49.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.764+0000 7fc161ffb700 1 -- 172.21.15.6:0/3799162477 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc15c0052b0 con 0x7fc17c0fcef0 2024-02-09T21:56:49.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.765+0000 7fc163fff700 1 -- 172.21.15.6:0/3799162477 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fc17000db90 con 0x7fc17c0fcef0 2024-02-09T21:56:49.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.765+0000 7fc163fff700 1 --2- 172.21.15.6:0/3799162477 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fc1640692b0 0x7fc16406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:49.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.765+0000 7fc163fff700 1 -- 172.21.15.6:0/3799162477 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fc170086fb0 con 0x7fc17c0fcef0 2024-02-09T21:56:49.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.765+0000 7fc17affd700 1 --2- 172.21.15.6:0/3799162477 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fc1640692b0 0x7fc16406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:49.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.766+0000 7fc17affd700 1 --2- 172.21.15.6:0/3799162477 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fc1640692b0 0x7fc16406b760 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc16c00b7a0 tx=0x7fc16c009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:49.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:49.770+0000 7fc163fff700 1 -- 172.21.15.6:0/3799162477 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fc170056c50 con 0x7fc17c0fcef0 2024-02-09T21:56:50.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:50.017+0000 7fc161ffb700 1 -- 172.21.15.6:0/3799162477 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7fc15c005ae0 con 0x7fc17c0fcef0 2024-02-09T21:56:50.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:50.286+0000 7fc163fff700 1 -- 172.21.15.6:0/3799162477 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v12) v1 ==== 121+0+0 (secure 0 0 0) 0x7fc1700567e0 con 0x7fc17c0fcef0 2024-02-09T21:56:50.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:50.289+0000 7fc161ffb700 1 -- 172.21.15.6:0/3799162477 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fc1640692b0 msgr2=0x7fc16406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:50.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:50.289+0000 7fc161ffb700 1 --2- 172.21.15.6:0/3799162477 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fc1640692b0 0x7fc16406b760 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc16c00b7a0 tx=0x7fc16c009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:50.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:50.289+0000 7fc161ffb700 1 -- 172.21.15.6:0/3799162477 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc17c0fcef0 msgr2=0x7fc17c10d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:50.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:50.289+0000 7fc161ffb700 1 --2- 172.21.15.6:0/3799162477 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc17c0fcef0 0x7fc17c10d970 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc170002a80 tx=0x7fc170004600 comp rx=0 tx=0).stop 2024-02-09T21:56:50.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:50.289+0000 7fc161ffb700 1 -- 172.21.15.6:0/3799162477 shutdown_connections 2024-02-09T21:56:50.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:50.289+0000 7fc161ffb700 1 --2- 172.21.15.6:0/3799162477 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fc1640692b0 0x7fc16406b760 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:50.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:50.289+0000 7fc161ffb700 1 --2- 172.21.15.6:0/3799162477 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc17c0fcef0 0x7fc17c10d970 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:50.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:50.289+0000 7fc161ffb700 1 --2- 172.21.15.6:0/3799162477 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc17c0fe0e0 0x7fc17c10deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:50.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:50.289+0000 7fc161ffb700 1 -- 172.21.15.6:0/3799162477 >> 172.21.15.6:0/3799162477 conn(0x7fc17c0f8620 msgr2=0x7fc17c1013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:50.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:50.289+0000 7fc161ffb700 1 -- 172.21.15.6:0/3799162477 shutdown_connections 2024-02-09T21:56:50.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:50.289+0000 7fc161ffb700 1 -- 172.21.15.6:0/3799162477 wait complete. 2024-02-09T21:56:50.495 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:50 smithi090 ceph-mon[36996]: Deploying daemon mds.cephfs.smithi090.mipeaa on smithi090 2024-02-09T21:56:50.496 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:50 smithi090 ceph-mon[36996]: pgmap v179: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.6 KiB/s wr, 8 op/s 2024-02-09T21:56:50.496 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:50 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/3799162477' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-02-09T21:56:50.604 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:50 smithi006 ceph-mon[29258]: Deploying daemon mds.cephfs.smithi090.mipeaa on smithi090 2024-02-09T21:56:50.604 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:50 smithi006 ceph-mon[29258]: pgmap v179: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.6 KiB/s wr, 8 op/s 2024-02-09T21:56:50.605 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:50 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/3799162477' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-02-09T21:56:50.990 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-09T21:56:51.001 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi006.front.sepia.ceph.com 2024-02-09T21:56:51.001 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-02-09T21:56:51.280 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:56:51.600 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:51 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/3799162477' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-02-09T21:56:51.600 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:51 smithi006 ceph-mon[29258]: fsmap cephfs:2 {0=cephfs.smithi006.ypgiqs=up:active,1=cephfs.smithi090.cgsuda=up:active} 1 up:standby 2024-02-09T21:56:51.601 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:51 smithi006 ceph-mon[29258]: fsmap cephfs:2 {0=cephfs.smithi006.ypgiqs=up:active,1=cephfs.smithi090.cgsuda=up:active} 1 up:standby-replay 2024-02-09T21:56:51.601 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:51 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:51.601 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:51 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:51.601 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:51 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:51.601 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:51 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:51.601 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:51 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:51.601 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:51 smithi006 ceph-mon[29258]: pgmap v180: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s wr, 8 op/s 2024-02-09T21:56:51.601 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:51 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:56:51.631 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:51 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/3799162477' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-02-09T21:56:51.631 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:51 smithi090 ceph-mon[36996]: fsmap cephfs:2 {0=cephfs.smithi006.ypgiqs=up:active,1=cephfs.smithi090.cgsuda=up:active} 1 up:standby 2024-02-09T21:56:51.631 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:51 smithi090 ceph-mon[36996]: fsmap cephfs:2 {0=cephfs.smithi006.ypgiqs=up:active,1=cephfs.smithi090.cgsuda=up:active} 1 up:standby-replay 2024-02-09T21:56:51.631 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:51 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:51.631 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:51 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:51.632 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:51 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:51.632 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:51 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:51.632 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:51 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:51.632 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:51 smithi090 ceph-mon[36996]: pgmap v180: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s wr, 8 op/s 2024-02-09T21:56:51.632 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:51 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:56:52.454 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:52 smithi090 ceph-mon[36996]: mds.? [v2:172.21.15.90:6826/1908638886,v1:172.21.15.90:6827/1908638886] up:boot 2024-02-09T21:56:52.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:52 smithi090 ceph-mon[36996]: mds.? [v2:172.21.15.6:6826/669315024,v1:172.21.15.6:6827/669315024] up:active 2024-02-09T21:56:52.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:52 smithi090 ceph-mon[36996]: fsmap cephfs:2 {0=cephfs.smithi006.ypgiqs=up:active,1=cephfs.smithi090.cgsuda=up:active} 1 up:standby-replay 1 up:standby 2024-02-09T21:56:52.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:52 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi090.mipeaa"}]: dispatch 2024-02-09T21:56:52.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:52 smithi090 ceph-mon[36996]: fsmap cephfs:2 {0=cephfs.smithi006.ypgiqs=up:active,1=cephfs.smithi090.cgsuda=up:active} 2 up:standby-replay 2024-02-09T21:56:52.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:52 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:52.521 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:52 smithi006 ceph-mon[29258]: mds.? [v2:172.21.15.90:6826/1908638886,v1:172.21.15.90:6827/1908638886] up:boot 2024-02-09T21:56:52.521 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:52 smithi006 ceph-mon[29258]: mds.? [v2:172.21.15.6:6826/669315024,v1:172.21.15.6:6827/669315024] up:active 2024-02-09T21:56:52.521 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:52 smithi006 ceph-mon[29258]: fsmap cephfs:2 {0=cephfs.smithi006.ypgiqs=up:active,1=cephfs.smithi090.cgsuda=up:active} 1 up:standby-replay 1 up:standby 2024-02-09T21:56:52.521 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:52 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi090.mipeaa"}]: dispatch 2024-02-09T21:56:52.521 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:52 smithi006 ceph-mon[29258]: fsmap cephfs:2 {0=cephfs.smithi006.ypgiqs=up:active,1=cephfs.smithi090.cgsuda=up:active} 2 up:standby-replay 2024-02-09T21:56:52.521 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:52 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:53.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.320+0000 7f9bd58a7700 1 -- 172.21.15.6:0/693687828 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bd00fb220 msgr2=0x7f9bd00fb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:53.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.320+0000 7f9bd58a7700 1 --2- 172.21.15.6:0/693687828 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bd00fb220 0x7f9bd00fb630 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0004940 tx=0x7f9bc0009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:53.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.320+0000 7f9bd58a7700 1 -- 172.21.15.6:0/693687828 shutdown_connections 2024-02-09T21:56:53.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.320+0000 7f9bd58a7700 1 --2- 172.21.15.6:0/693687828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9bd00fe2b0 0x7f9bd00fe700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:53.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.320+0000 7f9bd58a7700 1 --2- 172.21.15.6:0/693687828 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bd00fb220 0x7f9bd00fb630 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:53.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.320+0000 7f9bd58a7700 1 -- 172.21.15.6:0/693687828 >> 172.21.15.6:0/693687828 conn(0x7f9bd00f6ca0 msgr2=0x7f9bd00f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:53.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.320+0000 7f9bd58a7700 1 -- 172.21.15.6:0/693687828 shutdown_connections 2024-02-09T21:56:53.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.320+0000 7f9bd58a7700 1 -- 172.21.15.6:0/693687828 wait complete. 2024-02-09T21:56:53.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.321+0000 7f9bd58a7700 1 Processor -- start 2024-02-09T21:56:53.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.321+0000 7f9bd58a7700 1 -- start start 2024-02-09T21:56:53.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.321+0000 7f9bd58a7700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bd00fb220 0x7f9bd010da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:53.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.321+0000 7f9bd58a7700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9bd00fe2b0 0x7f9bd010df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:53.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.321+0000 7f9bd58a7700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd010e5f0 con 0x7f9bd00fb220 2024-02-09T21:56:53.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.321+0000 7f9bd58a7700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd010ee30 con 0x7f9bd00fe2b0 2024-02-09T21:56:53.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.321+0000 7f9bceffd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bd00fb220 0x7f9bd010da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:53.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.321+0000 7f9bceffd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bd00fb220 0x7f9bd010da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44172/0 (socket says 172.21.15.6:44172) 2024-02-09T21:56:53.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.321+0000 7f9bceffd700 1 -- 172.21.15.6:0/1469895729 learned_addr learned my addr 172.21.15.6:0/1469895729 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:56:53.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.321+0000 7f9bce7fc700 1 --2- 172.21.15.6:0/1469895729 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9bd00fe2b0 0x7f9bd010df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:53.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.321+0000 7f9bceffd700 1 -- 172.21.15.6:0/1469895729 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9bd00fe2b0 msgr2=0x7f9bd010df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:53.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.321+0000 7f9bceffd700 1 --2- 172.21.15.6:0/1469895729 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9bd00fe2b0 0x7f9bd010df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:53.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.321+0000 7f9bceffd700 1 -- 172.21.15.6:0/1469895729 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc0005040 con 0x7f9bd00fb220 2024-02-09T21:56:53.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.322+0000 7f9bceffd700 1 --2- 172.21.15.6:0/1469895729 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bd00fb220 0x7f9bd010da20 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0009150 tx=0x7f9bc0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:53.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.322+0000 7f9bb77fe700 1 -- 172.21.15.6:0/1469895729 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bc000e040 con 0x7f9bd00fb220 2024-02-09T21:56:53.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.322+0000 7f9bb77fe700 1 -- 172.21.15.6:0/1469895729 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9bc0009af0 con 0x7f9bd00fb220 2024-02-09T21:56:53.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.322+0000 7f9bd58a7700 1 -- 172.21.15.6:0/1469895729 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd010f0b0 con 0x7f9bd00fb220 2024-02-09T21:56:53.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.322+0000 7f9bb77fe700 1 -- 172.21.15.6:0/1469895729 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bc001c630 con 0x7f9bd00fb220 2024-02-09T21:56:53.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.322+0000 7f9bd58a7700 1 -- 172.21.15.6:0/1469895729 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd010f5f0 con 0x7f9bd00fb220 2024-02-09T21:56:53.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.323+0000 7f9bd58a7700 1 -- 172.21.15.6:0/1469895729 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bd004bed0 con 0x7f9bd00fb220 2024-02-09T21:56:53.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.323+0000 7f9bb77fe700 1 -- 172.21.15.6:0/1469895729 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f9bc000bbf0 con 0x7f9bd00fb220 2024-02-09T21:56:53.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.324+0000 7f9bb77fe700 1 --2- 172.21.15.6:0/1469895729 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f9bb80692a0 0x7f9bb806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:53.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.324+0000 7f9bb77fe700 1 -- 172.21.15.6:0/1469895729 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f9bc0088390 con 0x7f9bd00fb220 2024-02-09T21:56:53.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.324+0000 7f9bce7fc700 1 --2- 172.21.15.6:0/1469895729 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f9bb80692a0 0x7f9bb806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:53.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.324+0000 7f9bce7fc700 1 --2- 172.21.15.6:0/1469895729 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f9bb80692a0 0x7f9bb806b750 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4009bf0 tx=0x7f9bc40091a0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:53.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.328+0000 7f9bb77fe700 1 -- 172.21.15.6:0/1469895729 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9bc00581f0 con 0x7f9bd00fb220 2024-02-09T21:56:53.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:53.574+0000 7f9bd58a7700 1 -- 172.21.15.6:0/1469895729 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9bd0105ac0 con 0x7f9bd00fb220 2024-02-09T21:56:54.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:53 smithi006 ceph-mon[29258]: pgmap v181: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s rd, 2.0 KiB/s wr, 8 op/s 2024-02-09T21:56:54.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:53 smithi006 ceph-mon[29258]: mds.? [v2:172.21.15.6:6828/3025444869,v1:172.21.15.6:6829/3025444869] up:standby-replay 2024-02-09T21:56:54.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:53 smithi006 ceph-mon[29258]: fsmap cephfs:2 {0=cephfs.smithi006.ypgiqs=up:active,1=cephfs.smithi090.cgsuda=up:active} 2 up:standby-replay 2024-02-09T21:56:54.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:53 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:54.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:53 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:54.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:53 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/1469895729' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T21:56:54.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:53 smithi090 ceph-mon[36996]: pgmap v181: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s rd, 2.0 KiB/s wr, 8 op/s 2024-02-09T21:56:54.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:53 smithi090 ceph-mon[36996]: mds.? [v2:172.21.15.6:6828/3025444869,v1:172.21.15.6:6829/3025444869] up:standby-replay 2024-02-09T21:56:54.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:53 smithi090 ceph-mon[36996]: fsmap cephfs:2 {0=cephfs.smithi006.ypgiqs=up:active,1=cephfs.smithi090.cgsuda=up:active} 2 up:standby-replay 2024-02-09T21:56:54.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:53 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:54.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:53 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:54.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:53 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/1469895729' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T21:56:54.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:54.296+0000 7f9bb77fe700 1 -- 172.21.15.6:0/1469895729 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v17) v1 ==== 168+0+0 (secure 0 0 0) 0x7f9bc0057d80 con 0x7f9bd00fb220 2024-02-09T21:56:54.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:54.298+0000 7f9bd58a7700 1 -- 172.21.15.6:0/1469895729 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f9bb80692a0 msgr2=0x7f9bb806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:54.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:54.298+0000 7f9bd58a7700 1 --2- 172.21.15.6:0/1469895729 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f9bb80692a0 0x7f9bb806b750 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4009bf0 tx=0x7f9bc40091a0 comp rx=0 tx=0).stop 2024-02-09T21:56:54.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:54.298+0000 7f9bd58a7700 1 -- 172.21.15.6:0/1469895729 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bd00fb220 msgr2=0x7f9bd010da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:54.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:54.298+0000 7f9bd58a7700 1 --2- 172.21.15.6:0/1469895729 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bd00fb220 0x7f9bd010da20 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0009150 tx=0x7f9bc0009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:54.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:54.299+0000 7f9bd58a7700 1 -- 172.21.15.6:0/1469895729 shutdown_connections 2024-02-09T21:56:54.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:54.299+0000 7f9bd58a7700 1 --2- 172.21.15.6:0/1469895729 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f9bb80692a0 0x7f9bb806b750 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:54.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:54.299+0000 7f9bd58a7700 1 --2- 172.21.15.6:0/1469895729 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bd00fb220 0x7f9bd010da20 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:54.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:54.299+0000 7f9bd58a7700 1 --2- 172.21.15.6:0/1469895729 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9bd00fe2b0 0x7f9bd010df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:54.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:54.299+0000 7f9bd58a7700 1 -- 172.21.15.6:0/1469895729 >> 172.21.15.6:0/1469895729 conn(0x7f9bd00f6ca0 msgr2=0x7f9bd01014e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:54.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:54.299+0000 7f9bd58a7700 1 -- 172.21.15.6:0/1469895729 shutdown_connections 2024-02-09T21:56:54.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:54.299+0000 7f9bd58a7700 1 -- 172.21.15.6:0/1469895729 wait complete. 2024-02-09T21:56:54.303 INFO:teuthology.orchestra.run.smithi006.stderr:inline data enabled 2024-02-09T21:56:55.046 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:54 smithi006 ceph-mon[29258]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-02-09T21:56:55.046 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:54 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/1469895729' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-02-09T21:56:55.046 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:54 smithi006 ceph-mon[29258]: mds.? [v2:172.21.15.90:6824/2739488453,v1:172.21.15.90:6825/2739488453] up:active 2024-02-09T21:56:55.046 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:54 smithi006 ceph-mon[29258]: fsmap cephfs:2 {0=cephfs.smithi006.ypgiqs=up:active,1=cephfs.smithi090.cgsuda=up:active} 2 up:standby-replay 2024-02-09T21:56:55.071 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-09T21:56:55.082 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi006.front.sepia.ceph.com 2024-02-09T21:56:55.082 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- bash -c 'ceph fs dump' 2024-02-09T21:56:55.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:54 smithi090 ceph-mon[36996]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-02-09T21:56:55.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:54 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/1469895729' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-02-09T21:56:55.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:54 smithi090 ceph-mon[36996]: mds.? [v2:172.21.15.90:6824/2739488453,v1:172.21.15.90:6825/2739488453] up:active 2024-02-09T21:56:55.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:54 smithi090 ceph-mon[36996]: fsmap cephfs:2 {0=cephfs.smithi006.ypgiqs=up:active,1=cephfs.smithi090.cgsuda=up:active} 2 up:standby-replay 2024-02-09T21:56:55.501 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:56:56.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:55 smithi006 ceph-mon[29258]: pgmap v182: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-02-09T21:56:56.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:55 smithi006 ceph-mon[29258]: mds.? [v2:172.21.15.90:6826/1908638886,v1:172.21.15.90:6827/1908638886] up:standby-replay 2024-02-09T21:56:56.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:55 smithi006 ceph-mon[29258]: fsmap cephfs:2 {0=cephfs.smithi006.ypgiqs=up:active,1=cephfs.smithi090.cgsuda=up:active} 2 up:standby-replay 2024-02-09T21:56:56.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:55 smithi090 ceph-mon[36996]: pgmap v182: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-02-09T21:56:56.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:55 smithi090 ceph-mon[36996]: mds.? [v2:172.21.15.90:6826/1908638886,v1:172.21.15.90:6827/1908638886] up:standby-replay 2024-02-09T21:56:56.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:55 smithi090 ceph-mon[36996]: fsmap cephfs:2 {0=cephfs.smithi006.ypgiqs=up:active,1=cephfs.smithi090.cgsuda=up:active} 2 up:standby-replay 2024-02-09T21:56:57.954 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:57 smithi090 ceph-mon[36996]: pgmap v183: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-02-09T21:56:57.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:57 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:57.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:57 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:56:58.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.069+0000 7f7ac8d51700 1 -- 172.21.15.6:0/2930165392 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ac40663c0 msgr2=0x7f7ac40ffb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:58.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.069+0000 7f7ac8d51700 1 --2- 172.21.15.6:0/2930165392 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ac40663c0 0x7f7ac40ffb10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8004940 tx=0x7f7ab8009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:58.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.069+0000 7f7ac8d51700 1 -- 172.21.15.6:0/2930165392 shutdown_connections 2024-02-09T21:56:58.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.069+0000 7f7ac8d51700 1 --2- 172.21.15.6:0/2930165392 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ac41000e0 0x7f7ac4102520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:58.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.069+0000 7f7ac8d51700 1 --2- 172.21.15.6:0/2930165392 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ac40663c0 0x7f7ac40ffb10 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:58.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.069+0000 7f7ac8d51700 1 -- 172.21.15.6:0/2930165392 >> 172.21.15.6:0/2930165392 conn(0x7f7ac40f74e0 msgr2=0x7f7ac40f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:58.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.069+0000 7f7ac8d51700 1 -- 172.21.15.6:0/2930165392 shutdown_connections 2024-02-09T21:56:58.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.069+0000 7f7ac8d51700 1 -- 172.21.15.6:0/2930165392 wait complete. 2024-02-09T21:56:58.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.069+0000 7f7ac8d51700 1 Processor -- start 2024-02-09T21:56:58.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.069+0000 7f7ac8d51700 1 -- start start 2024-02-09T21:56:58.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.070+0000 7f7ac8d51700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ac40663c0 0x7f7ac40fd590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:58.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.070+0000 7f7ac8d51700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ac41000e0 0x7f7ac40fba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:58.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.070+0000 7f7ac8d51700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac40fc010 con 0x7f7ac41000e0 2024-02-09T21:56:58.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.070+0000 7f7ac8d51700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac40fc180 con 0x7f7ac40663c0 2024-02-09T21:56:58.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.070+0000 7f7ac1d9b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ac41000e0 0x7f7ac40fba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:58.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.070+0000 7f7ac1d9b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ac41000e0 0x7f7ac40fba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57414/0 (socket says 172.21.15.6:57414) 2024-02-09T21:56:58.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.070+0000 7f7ac1d9b700 1 -- 172.21.15.6:0/1231074871 learned_addr learned my addr 172.21.15.6:0/1231074871 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:56:58.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.070+0000 7f7ac259c700 1 --2- 172.21.15.6:0/1231074871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ac40663c0 0x7f7ac40fd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:58.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.070+0000 7f7ac1d9b700 1 -- 172.21.15.6:0/1231074871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ac40663c0 msgr2=0x7f7ac40fd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:58.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.070+0000 7f7ac1d9b700 1 --2- 172.21.15.6:0/1231074871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ac40663c0 0x7f7ac40fd590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:58.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.070+0000 7f7ac1d9b700 1 -- 172.21.15.6:0/1231074871 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab8005040 con 0x7f7ac41000e0 2024-02-09T21:56:58.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.070+0000 7f7ac1d9b700 1 --2- 172.21.15.6:0/1231074871 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ac41000e0 0x7f7ac40fba10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7ab400c890 tx=0x7f7ab40044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:58.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.070+0000 7f7ab2ffd700 1 -- 172.21.15.6:0/1231074871 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ab400a7d0 con 0x7f7ac41000e0 2024-02-09T21:56:58.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.070+0000 7f7ac8d51700 1 -- 172.21.15.6:0/1231074871 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ac40fc460 con 0x7f7ac41000e0 2024-02-09T21:56:58.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.070+0000 7f7ab2ffd700 1 -- 172.21.15.6:0/1231074871 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ab400ae00 con 0x7f7ac41000e0 2024-02-09T21:56:58.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.070+0000 7f7ab2ffd700 1 -- 172.21.15.6:0/1231074871 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ab40113a0 con 0x7f7ac41000e0 2024-02-09T21:56:58.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.070+0000 7f7ac8d51700 1 -- 172.21.15.6:0/1231074871 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ac4115980 con 0x7f7ac41000e0 2024-02-09T21:56:58.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.071+0000 7f7ac8d51700 1 -- 172.21.15.6:0/1231074871 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ac4107a70 con 0x7f7ac41000e0 2024-02-09T21:56:58.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.072+0000 7f7ab2ffd700 1 -- 172.21.15.6:0/1231074871 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f7ab4012400 con 0x7f7ac41000e0 2024-02-09T21:56:58.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.072+0000 7f7ab2ffd700 1 --2- 172.21.15.6:0/1231074871 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f7aac0692a0 0x7f7aac06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:56:58.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.072+0000 7f7ac259c700 1 --2- 172.21.15.6:0/1231074871 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f7aac0692a0 0x7f7aac06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:56:58.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.072+0000 7f7ab2ffd700 1 -- 172.21.15.6:0/1231074871 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f7ab4087280 con 0x7f7ac41000e0 2024-02-09T21:56:58.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.073+0000 7f7ac259c700 1 --2- 172.21.15.6:0/1231074871 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f7aac0692a0 0x7f7aac06b750 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8000c00 tx=0x7f7ab8009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:56:58.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.077+0000 7f7ab2ffd700 1 -- 172.21.15.6:0/1231074871 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7ab4057040 con 0x7f7ac41000e0 2024-02-09T21:56:58.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:57 smithi006 ceph-mon[29258]: pgmap v183: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-02-09T21:56:58.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:57 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:56:58.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:57 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:56:58.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.315+0000 7f7ac8d51700 1 -- 172.21.15.6:0/1231074871 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7ac402b390 con 0x7f7ac41000e0 2024-02-09T21:56:58.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.315+0000 7f7ab2ffd700 1 -- 172.21.15.6:0/1231074871 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1822 (secure 0 0 0) 0x7f7ab4056bd0 con 0x7f7ac41000e0 2024-02-09T21:56:58.319 INFO:teuthology.orchestra.run.smithi006.stdout:e18 2024-02-09T21:56:58.319 INFO:teuthology.orchestra.run.smithi006.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-09T21:56:58.319 INFO:teuthology.orchestra.run.smithi006.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-09T21:56:58.319 INFO:teuthology.orchestra.run.smithi006.stdout:legacy client fscid: 1 2024-02-09T21:56:58.320 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:56:58.320 INFO:teuthology.orchestra.run.smithi006.stdout:Filesystem 'cephfs' (1) 2024-02-09T21:56:58.320 INFO:teuthology.orchestra.run.smithi006.stdout:fs_name cephfs 2024-02-09T21:56:58.320 INFO:teuthology.orchestra.run.smithi006.stdout:epoch 18 2024-02-09T21:56:58.320 INFO:teuthology.orchestra.run.smithi006.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-09T21:56:58.320 INFO:teuthology.orchestra.run.smithi006.stdout:created 2024-02-09T21:56:38.486224+0000 2024-02-09T21:56:58.320 INFO:teuthology.orchestra.run.smithi006.stdout:modified 2024-02-09T21:56:55.295828+0000 2024-02-09T21:56:58.320 INFO:teuthology.orchestra.run.smithi006.stdout:tableserver 0 2024-02-09T21:56:58.321 INFO:teuthology.orchestra.run.smithi006.stdout:root 0 2024-02-09T21:56:58.321 INFO:teuthology.orchestra.run.smithi006.stdout:session_timeout 60 2024-02-09T21:56:58.321 INFO:teuthology.orchestra.run.smithi006.stdout:session_autoclose 300 2024-02-09T21:56:58.321 INFO:teuthology.orchestra.run.smithi006.stdout:max_file_size 1099511627776 2024-02-09T21:56:58.321 INFO:teuthology.orchestra.run.smithi006.stdout:required_client_features {} 2024-02-09T21:56:58.321 INFO:teuthology.orchestra.run.smithi006.stdout:last_failure 0 2024-02-09T21:56:58.321 INFO:teuthology.orchestra.run.smithi006.stdout:last_failure_osd_epoch 0 2024-02-09T21:56:58.321 INFO:teuthology.orchestra.run.smithi006.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-09T21:56:58.321 INFO:teuthology.orchestra.run.smithi006.stdout:max_mds 2 2024-02-09T21:56:58.321 INFO:teuthology.orchestra.run.smithi006.stdout:in 0,1 2024-02-09T21:56:58.321 INFO:teuthology.orchestra.run.smithi006.stdout:up {0=14458,1=24271} 2024-02-09T21:56:58.321 INFO:teuthology.orchestra.run.smithi006.stdout:failed 2024-02-09T21:56:58.321 INFO:teuthology.orchestra.run.smithi006.stdout:damaged 2024-02-09T21:56:58.321 INFO:teuthology.orchestra.run.smithi006.stdout:stopped 2024-02-09T21:56:58.321 INFO:teuthology.orchestra.run.smithi006.stdout:data_pools [3] 2024-02-09T21:56:58.321 INFO:teuthology.orchestra.run.smithi006.stdout:metadata_pool 2 2024-02-09T21:56:58.322 INFO:teuthology.orchestra.run.smithi006.stdout:inline_data enabled 2024-02-09T21:56:58.322 INFO:teuthology.orchestra.run.smithi006.stdout:balancer 2024-02-09T21:56:58.322 INFO:teuthology.orchestra.run.smithi006.stdout:standby_count_wanted 1 2024-02-09T21:56:58.322 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.cephfs.smithi006.ypgiqs{0:14458} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.6:6826/669315024,v1:172.21.15.6:6827/669315024] compat {c=[1],r=[1],i=[7ff]}] 2024-02-09T21:56:58.322 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.cephfs.smithi006.qyamnu{0:14472} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.6:6828/3025444869,v1:172.21.15.6:6829/3025444869] compat {c=[1],r=[1],i=[7ff]}] 2024-02-09T21:56:58.322 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.cephfs.smithi090.cgsuda{1:24271} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.90:6824/2739488453,v1:172.21.15.90:6825/2739488453] compat {c=[1],r=[1],i=[7ff]}] 2024-02-09T21:56:58.322 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.cephfs.smithi090.mipeaa{1:24287} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.90:6826/1908638886,v1:172.21.15.90:6827/1908638886] compat {c=[1],r=[1],i=[7ff]}] 2024-02-09T21:56:58.322 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:56:58.322 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:56:58.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.317+0000 7f7ac8d51700 1 -- 172.21.15.6:0/1231074871 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f7aac0692a0 msgr2=0x7f7aac06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:58.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.317+0000 7f7ac8d51700 1 --2- 172.21.15.6:0/1231074871 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f7aac0692a0 0x7f7aac06b750 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8000c00 tx=0x7f7ab8009e30 comp rx=0 tx=0).stop 2024-02-09T21:56:58.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.317+0000 7f7ac8d51700 1 -- 172.21.15.6:0/1231074871 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ac41000e0 msgr2=0x7f7ac40fba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:56:58.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.317+0000 7f7ac8d51700 1 --2- 172.21.15.6:0/1231074871 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ac41000e0 0x7f7ac40fba10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7ab400c890 tx=0x7f7ab40044c0 comp rx=0 tx=0).stop 2024-02-09T21:56:58.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.317+0000 7f7ac8d51700 1 -- 172.21.15.6:0/1231074871 shutdown_connections 2024-02-09T21:56:58.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.317+0000 7f7ac8d51700 1 --2- 172.21.15.6:0/1231074871 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f7aac0692a0 0x7f7aac06b750 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:58.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.317+0000 7f7ac8d51700 1 --2- 172.21.15.6:0/1231074871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ac40663c0 0x7f7ac40fd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:58.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.317+0000 7f7ac8d51700 1 --2- 172.21.15.6:0/1231074871 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ac41000e0 0x7f7ac40fba10 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:56:58.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.317+0000 7f7ac8d51700 1 -- 172.21.15.6:0/1231074871 >> 172.21.15.6:0/1231074871 conn(0x7f7ac40f74e0 msgr2=0x7f7ac40f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:56:58.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.317+0000 7f7ac8d51700 1 -- 172.21.15.6:0/1231074871 shutdown_connections 2024-02-09T21:56:58.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:56:58.317+0000 7f7ac8d51700 1 -- 172.21.15.6:0/1231074871 wait complete. 2024-02-09T21:56:58.324 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 18 2024-02-09T21:56:58.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:56:58 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/1231074871' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-09T21:56:58.998 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-02-09T21:56:59.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:56:58 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/1231074871' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-09T21:56:59.294 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:57:00.393 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:00 smithi006 ceph-mon[29258]: pgmap v184: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 853 B/s wr, 7 op/s 2024-02-09T21:57:00.394 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:00 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:57:00.394 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:00 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:57:00.394 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:00 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:57:00.394 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:00 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:57:00.394 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:00 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:57:00.394 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:00 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:57:00.552 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:00 smithi090 ceph-mon[36996]: pgmap v184: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 853 B/s wr, 7 op/s 2024-02-09T21:57:00.552 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:00 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:57:00.552 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:00 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:57:00.552 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:00 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:57:00.552 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:00 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:57:00.552 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:00 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:57:00.552 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:00 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:57:00.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.906+0000 7f7b8fe6d700 1 -- 172.21.15.6:0/2292244573 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b880fe0e0 msgr2=0x7f7b880fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:00.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.906+0000 7f7b8fe6d700 1 --2- 172.21.15.6:0/2292244573 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b880fe0e0 0x7f7b880fe550 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c004940 tx=0x7f7b7c009e30 comp rx=0 tx=0).stop 2024-02-09T21:57:00.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.906+0000 7f7b8fe6d700 1 -- 172.21.15.6:0/2292244573 shutdown_connections 2024-02-09T21:57:00.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.906+0000 7f7b8fe6d700 1 --2- 172.21.15.6:0/2292244573 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b880fe0e0 0x7f7b880fe550 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:00.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.906+0000 7f7b8fe6d700 1 --2- 172.21.15.6:0/2292244573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b880fcef0 0x7f7b880fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:00.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.906+0000 7f7b8fe6d700 1 -- 172.21.15.6:0/2292244573 >> 172.21.15.6:0/2292244573 conn(0x7f7b880f8620 msgr2=0x7f7b880faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:57:00.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.906+0000 7f7b8fe6d700 1 -- 172.21.15.6:0/2292244573 shutdown_connections 2024-02-09T21:57:00.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.906+0000 7f7b8fe6d700 1 -- 172.21.15.6:0/2292244573 wait complete. 2024-02-09T21:57:00.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.907+0000 7f7b8fe6d700 1 Processor -- start 2024-02-09T21:57:00.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.907+0000 7f7b8fe6d700 1 -- start start 2024-02-09T21:57:00.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.907+0000 7f7b8fe6d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b880fcef0 0x7f7b8810d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:00.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.907+0000 7f7b8fe6d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b880fe0e0 0x7f7b8810de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:00.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.907+0000 7f7b8fe6d700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8810e430 con 0x7f7b880fe0e0 2024-02-09T21:57:00.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.907+0000 7f7b8fe6d700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8810ecd0 con 0x7f7b880fcef0 2024-02-09T21:57:00.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.907+0000 7f7b8d408700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b880fe0e0 0x7f7b8810de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:00.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.907+0000 7f7b8d408700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b880fe0e0 0x7f7b8810de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57446/0 (socket says 172.21.15.6:57446) 2024-02-09T21:57:00.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.907+0000 7f7b8d408700 1 -- 172.21.15.6:0/3383206176 learned_addr learned my addr 172.21.15.6:0/3383206176 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:57:00.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.907+0000 7f7b8dc09700 1 --2- 172.21.15.6:0/3383206176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b880fcef0 0x7f7b8810d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:00.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.907+0000 7f7b8d408700 1 -- 172.21.15.6:0/3383206176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b880fcef0 msgr2=0x7f7b8810d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:00.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.907+0000 7f7b8d408700 1 --2- 172.21.15.6:0/3383206176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b880fcef0 0x7f7b8810d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:00.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.907+0000 7f7b8d408700 1 -- 172.21.15.6:0/3383206176 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b7c005040 con 0x7f7b880fe0e0 2024-02-09T21:57:00.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.907+0000 7f7b8d408700 1 --2- 172.21.15.6:0/3383206176 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b880fe0e0 0x7f7b8810de30 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c000c00 tx=0x7f7b7c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:57:00.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.908+0000 7f7b867fc700 1 -- 172.21.15.6:0/3383206176 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b7c00e070 con 0x7f7b880fe0e0 2024-02-09T21:57:00.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.908+0000 7f7b867fc700 1 -- 172.21.15.6:0/3383206176 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b7c01c400 con 0x7f7b880fe0e0 2024-02-09T21:57:00.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.908+0000 7f7b867fc700 1 -- 172.21.15.6:0/3383206176 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b7c00b650 con 0x7f7b880fe0e0 2024-02-09T21:57:00.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.908+0000 7f7b8fe6d700 1 -- 172.21.15.6:0/3383206176 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b8810ef50 con 0x7f7b880fe0e0 2024-02-09T21:57:00.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.908+0000 7f7b8fe6d700 1 -- 172.21.15.6:0/3383206176 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b8810f370 con 0x7f7b880fe0e0 2024-02-09T21:57:00.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.909+0000 7f7b8fe6d700 1 -- 172.21.15.6:0/3383206176 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b8804bed0 con 0x7f7b880fe0e0 2024-02-09T21:57:00.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.909+0000 7f7b867fc700 1 -- 172.21.15.6:0/3383206176 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f7b7c01ca50 con 0x7f7b880fe0e0 2024-02-09T21:57:00.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.910+0000 7f7b867fc700 1 --2- 172.21.15.6:0/3383206176 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f7b700692a0 0x7f7b7006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:00.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.910+0000 7f7b867fc700 1 -- 172.21.15.6:0/3383206176 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f7b7c0887e0 con 0x7f7b880fe0e0 2024-02-09T21:57:00.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.910+0000 7f7b8dc09700 1 --2- 172.21.15.6:0/3383206176 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f7b700692a0 0x7f7b7006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:00.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.910+0000 7f7b8dc09700 1 --2- 172.21.15.6:0/3383206176 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f7b700692a0 0x7f7b7006b750 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7b880fcc30 tx=0x7f7b8000b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:57:00.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:00.914+0000 7f7b867fc700 1 -- 172.21.15.6:0/3383206176 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7b7c0585a0 con 0x7f7b880fe0e0 2024-02-09T21:57:01.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:01.152+0000 7f7b8fe6d700 1 -- 172.21.15.6:0/3383206176 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b88105a30 con 0x7f7b880fe0e0 2024-02-09T21:57:01.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:01.153+0000 7f7b867fc700 1 -- 172.21.15.6:0/3383206176 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4708 (secure 0 0 0) 0x7f7b88105a30 con 0x7f7b880fe0e0 2024-02-09T21:57:01.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:01.155+0000 7f7b8fe6d700 1 -- 172.21.15.6:0/3383206176 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f7b700692a0 msgr2=0x7f7b7006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:01.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:01.155+0000 7f7b8fe6d700 1 --2- 172.21.15.6:0/3383206176 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f7b700692a0 0x7f7b7006b750 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7b880fcc30 tx=0x7f7b8000b040 comp rx=0 tx=0).stop 2024-02-09T21:57:01.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:01.155+0000 7f7b8fe6d700 1 -- 172.21.15.6:0/3383206176 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b880fe0e0 msgr2=0x7f7b8810de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:01.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:01.155+0000 7f7b8fe6d700 1 --2- 172.21.15.6:0/3383206176 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b880fe0e0 0x7f7b8810de30 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c000c00 tx=0x7f7b7c009e30 comp rx=0 tx=0).stop 2024-02-09T21:57:01.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:01.155+0000 7f7b8fe6d700 1 -- 172.21.15.6:0/3383206176 shutdown_connections 2024-02-09T21:57:01.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:01.155+0000 7f7b8fe6d700 1 --2- 172.21.15.6:0/3383206176 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f7b700692a0 0x7f7b7006b750 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:01.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:01.155+0000 7f7b8fe6d700 1 --2- 172.21.15.6:0/3383206176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b880fcef0 0x7f7b8810d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:01.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:01.155+0000 7f7b8fe6d700 1 --2- 172.21.15.6:0/3383206176 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b880fe0e0 0x7f7b8810de30 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:01.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:01.155+0000 7f7b8fe6d700 1 -- 172.21.15.6:0/3383206176 >> 172.21.15.6:0/3383206176 conn(0x7f7b880f8620 msgr2=0x7f7b881013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:57:01.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:01.155+0000 7f7b8fe6d700 1 -- 172.21.15.6:0/3383206176 shutdown_connections 2024-02-09T21:57:01.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:01.155+0000 7f7b8fe6d700 1 -- 172.21.15.6:0/3383206176 wait complete. 2024-02-09T21:57:01.160 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 18 2024-02-09T21:57:01.169 INFO:teuthology.orchestra.run.smithi006.stdout:true 2024-02-09T21:57:01.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:01 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/3383206176' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T21:57:01.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:01 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/3383206176' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T21:57:02.165 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-02-09T21:57:02.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:02 smithi006 ceph-mon[29258]: pgmap v185: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-09T21:57:02.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:02 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:57:02.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:02 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:57:02.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:02 smithi090 ceph-mon[36996]: pgmap v185: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-09T21:57:02.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:02 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:57:02.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:02 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:57:03.076 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:57:04.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:04 smithi006 ceph-mon[29258]: pgmap v186: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 4 op/s 2024-02-09T21:57:04.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:04 smithi090 ceph-mon[36996]: pgmap v186: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 4 op/s 2024-02-09T21:57:05.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.947+0000 7f0e2efc8700 1 -- 172.21.15.6:0/1641333597 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e280fe0e0 msgr2=0x7f0e280fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:05.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.947+0000 7f0e2efc8700 1 --2- 172.21.15.6:0/1641333597 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e280fe0e0 0x7f0e280fe550 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0e18004940 tx=0x7f0e18009e30 comp rx=0 tx=0).stop 2024-02-09T21:57:05.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.947+0000 7f0e2efc8700 1 -- 172.21.15.6:0/1641333597 shutdown_connections 2024-02-09T21:57:05.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.947+0000 7f0e2efc8700 1 --2- 172.21.15.6:0/1641333597 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e280fe0e0 0x7f0e280fe550 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:05.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.947+0000 7f0e2efc8700 1 --2- 172.21.15.6:0/1641333597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e280fcef0 0x7f0e280fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:05.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.947+0000 7f0e2efc8700 1 -- 172.21.15.6:0/1641333597 >> 172.21.15.6:0/1641333597 conn(0x7f0e280f8620 msgr2=0x7f0e280faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:57:05.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.947+0000 7f0e2efc8700 1 -- 172.21.15.6:0/1641333597 shutdown_connections 2024-02-09T21:57:05.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.947+0000 7f0e2efc8700 1 -- 172.21.15.6:0/1641333597 wait complete. 2024-02-09T21:57:05.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.948+0000 7f0e2efc8700 1 Processor -- start 2024-02-09T21:57:05.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.948+0000 7f0e2efc8700 1 -- start start 2024-02-09T21:57:05.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.948+0000 7f0e2efc8700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e280fcef0 0x7f0e2810d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:05.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.948+0000 7f0e2efc8700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e280fe0e0 0x7f0e2810dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:05.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.948+0000 7f0e2efc8700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2810e4a0 con 0x7f0e280fcef0 2024-02-09T21:57:05.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.948+0000 7f0e2efc8700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2810ed40 con 0x7f0e280fe0e0 2024-02-09T21:57:05.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.948+0000 7f0e2cd64700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e280fcef0 0x7f0e2810d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:05.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.948+0000 7f0e2cd64700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e280fcef0 0x7f0e2810d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57454/0 (socket says 172.21.15.6:57454) 2024-02-09T21:57:05.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.948+0000 7f0e2cd64700 1 -- 172.21.15.6:0/2317262160 learned_addr learned my addr 172.21.15.6:0/2317262160 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:57:05.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.949+0000 7f0e27fff700 1 --2- 172.21.15.6:0/2317262160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e280fe0e0 0x7f0e2810dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:05.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.949+0000 7f0e2cd64700 1 -- 172.21.15.6:0/2317262160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e280fe0e0 msgr2=0x7f0e2810dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:05.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.949+0000 7f0e2cd64700 1 --2- 172.21.15.6:0/2317262160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e280fe0e0 0x7f0e2810dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:05.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.949+0000 7f0e2cd64700 1 -- 172.21.15.6:0/2317262160 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e18005040 con 0x7f0e280fcef0 2024-02-09T21:57:05.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.949+0000 7f0e2cd64700 1 --2- 172.21.15.6:0/2317262160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e280fcef0 0x7f0e2810d960 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0e10002a80 tx=0x7f0e10004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:57:05.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.949+0000 7f0e257fa700 1 -- 172.21.15.6:0/2317262160 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e10007cc0 con 0x7f0e280fcef0 2024-02-09T21:57:05.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.949+0000 7f0e2efc8700 1 -- 172.21.15.6:0/2317262160 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e2810f020 con 0x7f0e280fcef0 2024-02-09T21:57:05.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.949+0000 7f0e257fa700 1 -- 172.21.15.6:0/2317262160 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e1000f040 con 0x7f0e280fcef0 2024-02-09T21:57:05.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.949+0000 7f0e257fa700 1 -- 172.21.15.6:0/2317262160 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e1000da30 con 0x7f0e280fcef0 2024-02-09T21:57:05.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.949+0000 7f0e2efc8700 1 -- 172.21.15.6:0/2317262160 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e2810f560 con 0x7f0e280fcef0 2024-02-09T21:57:05.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.950+0000 7f0e2efc8700 1 -- 172.21.15.6:0/2317262160 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e2804bed0 con 0x7f0e280fcef0 2024-02-09T21:57:05.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.950+0000 7f0e257fa700 1 -- 172.21.15.6:0/2317262160 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f0e1000db90 con 0x7f0e280fcef0 2024-02-09T21:57:05.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.951+0000 7f0e257fa700 1 --2- 172.21.15.6:0/2317262160 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f0e140692a0 0x7f0e1406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:05.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.951+0000 7f0e257fa700 1 -- 172.21.15.6:0/2317262160 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f0e10086f10 con 0x7f0e280fcef0 2024-02-09T21:57:05.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.951+0000 7f0e27fff700 1 --2- 172.21.15.6:0/2317262160 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f0e140692a0 0x7f0e1406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:05.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.951+0000 7f0e27fff700 1 --2- 172.21.15.6:0/2317262160 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f0e140692a0 0x7f0e1406b750 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0e1800b3b0 tx=0x7f0e18009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:57:05.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:05.956+0000 7f0e257fa700 1 -- 172.21.15.6:0/2317262160 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0e10056c50 con 0x7f0e280fcef0 2024-02-09T21:57:06.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:06.237+0000 7f0e2efc8700 1 -- 172.21.15.6:0/2317262160 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f0e2810f950 con 0x7f0e280fcef0 2024-02-09T21:57:06.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:06.238+0000 7f0e257fa700 1 -- 172.21.15.6:0/2317262160 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v18) v1 ==== 78+0+99 (secure 0 0 0) 0x7f0e100567e0 con 0x7f0e280fcef0 2024-02-09T21:57:06.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:06.240+0000 7f0e2efc8700 1 -- 172.21.15.6:0/2317262160 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f0e140692a0 msgr2=0x7f0e1406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:06.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:06.240+0000 7f0e2efc8700 1 --2- 172.21.15.6:0/2317262160 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f0e140692a0 0x7f0e1406b750 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0e1800b3b0 tx=0x7f0e18009e30 comp rx=0 tx=0).stop 2024-02-09T21:57:06.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:06.240+0000 7f0e2efc8700 1 -- 172.21.15.6:0/2317262160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e280fcef0 msgr2=0x7f0e2810d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:06.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:06.240+0000 7f0e2efc8700 1 --2- 172.21.15.6:0/2317262160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e280fcef0 0x7f0e2810d960 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0e10002a80 tx=0x7f0e10004600 comp rx=0 tx=0).stop 2024-02-09T21:57:06.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:06.240+0000 7f0e2efc8700 1 -- 172.21.15.6:0/2317262160 shutdown_connections 2024-02-09T21:57:06.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:06.240+0000 7f0e2efc8700 1 --2- 172.21.15.6:0/2317262160 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f0e140692a0 0x7f0e1406b750 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:06.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:06.240+0000 7f0e2efc8700 1 --2- 172.21.15.6:0/2317262160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e280fcef0 0x7f0e2810d960 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:06.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:06.240+0000 7f0e2efc8700 1 --2- 172.21.15.6:0/2317262160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e280fe0e0 0x7f0e2810dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:06.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:06.240+0000 7f0e2efc8700 1 -- 172.21.15.6:0/2317262160 >> 172.21.15.6:0/2317262160 conn(0x7f0e280f8620 msgr2=0x7f0e281013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:57:06.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:06.240+0000 7f0e2efc8700 1 -- 172.21.15.6:0/2317262160 shutdown_connections 2024-02-09T21:57:06.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:06.240+0000 7f0e2efc8700 1 -- 172.21.15.6:0/2317262160 wait complete. 2024-02-09T21:57:06.256 INFO:teuthology.orchestra.run.smithi006.stdout:true 2024-02-09T21:57:06.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:06 smithi006 ceph-mon[29258]: pgmap v187: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 3 op/s 2024-02-09T21:57:06.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:06 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/2317262160' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-02-09T21:57:06.704 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:06 smithi090 ceph-mon[36996]: pgmap v187: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 3 op/s 2024-02-09T21:57:06.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:06 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/2317262160' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-02-09T21:57:07.003 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-02-09T21:57:07.016 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-09T21:57:07.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.143+0000 7f2b32258640 1 -- 172.21.15.6:0/3787594505 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b2c0fc450 msgr2=0x7f2b2c0fc8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:07.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.143+0000 7f2b32258640 1 --2- 172.21.15.6:0/3787594505 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b2c0fc450 0x7f2b2c0fc8c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2b200048d0 tx=0x7f2b20009da0 comp rx=0 tx=0).stop 2024-02-09T21:57:07.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.143+0000 7f2b32258640 1 -- 172.21.15.6:0/3787594505 shutdown_connections 2024-02-09T21:57:07.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.143+0000 7f2b32258640 1 --2- 172.21.15.6:0/3787594505 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b2c0fc450 0x7f2b2c0fc8c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:07.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.143+0000 7f2b32258640 1 --2- 172.21.15.6:0/3787594505 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b2c0fb180 0x7f2b2c0fb590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:07.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.143+0000 7f2b32258640 1 -- 172.21.15.6:0/3787594505 >> 172.21.15.6:0/3787594505 conn(0x7f2b2c0f6940 msgr2=0x7f2b2c0f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:57:07.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.143+0000 7f2b32258640 1 -- 172.21.15.6:0/3787594505 shutdown_connections 2024-02-09T21:57:07.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.143+0000 7f2b32258640 1 -- 172.21.15.6:0/3787594505 wait complete. 2024-02-09T21:57:07.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.143+0000 7f2b32258640 1 Processor -- start 2024-02-09T21:57:07.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.144+0000 7f2b32258640 1 -- start start 2024-02-09T21:57:07.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.144+0000 7f2b32258640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b2c0fb180 0x7f2b2c10ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:07.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.144+0000 7f2b32258640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b2c0fc450 0x7f2b2c10bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:07.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.144+0000 7f2b32258640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c10c590 con 0x7f2b2c0fb180 2024-02-09T21:57:07.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.144+0000 7f2b32258640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c110fe0 con 0x7f2b2c0fc450 2024-02-09T21:57:07.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.144+0000 7f2b2b7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b2c0fb180 0x7f2b2c10ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:07.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.144+0000 7f2b2b7fe640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b2c0fb180 0x7f2b2c10ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57472/0 (socket says 172.21.15.6:57472) 2024-02-09T21:57:07.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.144+0000 7f2b2b7fe640 1 -- 172.21.15.6:0/2683967614 learned_addr learned my addr 172.21.15.6:0/2683967614 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:57:07.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.144+0000 7f2b2affd640 1 --2- 172.21.15.6:0/2683967614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b2c0fc450 0x7f2b2c10bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:07.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.144+0000 7f2b2b7fe640 1 -- 172.21.15.6:0/2683967614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b2c0fc450 msgr2=0x7f2b2c10bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:07.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.144+0000 7f2b2b7fe640 1 --2- 172.21.15.6:0/2683967614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b2c0fc450 0x7f2b2c10bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:07.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.144+0000 7f2b2b7fe640 1 -- 172.21.15.6:0/2683967614 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b20005040 con 0x7f2b2c0fb180 2024-02-09T21:57:07.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.144+0000 7f2b2affd640 1 --2- 172.21.15.6:0/2683967614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b2c0fc450 0x7f2b2c10bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T21:57:07.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.144+0000 7f2b2b7fe640 1 --2- 172.21.15.6:0/2683967614 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b2c0fb180 0x7f2b2c10ba00 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c00eb70 tx=0x7f2b1c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:57:07.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.145+0000 7f2b13fff640 1 -- 172.21.15.6:0/2683967614 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b1c007c30 con 0x7f2b2c0fb180 2024-02-09T21:57:07.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.145+0000 7f2b13fff640 1 -- 172.21.15.6:0/2683967614 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b1c00f040 con 0x7f2b2c0fb180 2024-02-09T21:57:07.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.145+0000 7f2b32258640 1 -- 172.21.15.6:0/2683967614 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b2c111120 con 0x7f2b2c0fb180 2024-02-09T21:57:07.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.145+0000 7f2b13fff640 1 -- 172.21.15.6:0/2683967614 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b1c009680 con 0x7f2b2c0fb180 2024-02-09T21:57:07.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.145+0000 7f2b32258640 1 -- 172.21.15.6:0/2683967614 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b2c111610 con 0x7f2b2c0fb180 2024-02-09T21:57:07.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.146+0000 7f2b32258640 1 -- 172.21.15.6:0/2683967614 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b2c103cd0 con 0x7f2b2c0fb180 2024-02-09T21:57:07.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.146+0000 7f2b13fff640 1 -- 172.21.15.6:0/2683967614 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f2b1c00cd40 con 0x7f2b2c0fb180 2024-02-09T21:57:07.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.147+0000 7f2b13fff640 1 --2- 172.21.15.6:0/2683967614 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f2b140692e0 0x7f2b1406b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:07.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.147+0000 7f2b2affd640 1 --2- 172.21.15.6:0/2683967614 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f2b140692e0 0x7f2b1406b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:07.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.147+0000 7f2b13fff640 1 -- 172.21.15.6:0/2683967614 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f2b1c086e70 con 0x7f2b2c0fb180 2024-02-09T21:57:07.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.147+0000 7f2b2affd640 1 --2- 172.21.15.6:0/2683967614 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f2b140692e0 0x7f2b1406b790 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2b2002bda0 tx=0x7f2b20009da0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:57:07.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.151+0000 7f2b13fff640 1 -- 172.21.15.6:0/2683967614 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2b1c04dbd0 con 0x7f2b2c0fb180 2024-02-09T21:57:07.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.333+0000 7f2b32258640 1 -- 172.21.15.6:0/2683967614 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b2c111dc0 con 0x7f2b2c0fb180 2024-02-09T21:57:07.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.334+0000 7f2b13fff640 1 -- 172.21.15.6:0/2683967614 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+14180 (secure 0 0 0) 0x7f2b1c0531b0 con 0x7f2b2c0fb180 2024-02-09T21:57:07.337 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:57:07.337 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":45,"fsid":"ee26d820-c794-11ee-95b8-87774f69a715","created":"2024-02-09T21:49:39.138061+0000","modified":"2024-02-09T21:56:40.490741+0000","last_up_change":"2024-02-09T21:55:22.274974+0000","last_in_change":"2024-02-09T21:55:03.608607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-09T21:54:09.146714+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-02-09T21:56:37.425420+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-02-09T21:56:37.727029+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c78f3c5b-14ab-40b7-bd39-52cb040b6532","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":4292395427},{"type":"v1","addr":"172.21.15.6:6803","nonce":4292395427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":4292395427},{"type":"v1","addr":"172.21.15.6:6805","nonce":4292395427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":4292395427},{"type":"v1","addr":"172.21.15.6:6809","nonce":4292395427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":4292395427},{"type":"v1","addr":"172.21.15.6:6807","nonce":4292395427}]},"public_addr":"172.21.15.6:6803/4292395427","cluster_addr":"172.21.15.6:6805/4292395427","heartbeat_back_addr":"172.21.15.6:6809/4292395427","heartbeat_front_addr":"172.21.15.6:6807/4292395427","state":["exists","up"]},{"osd":1,"uuid":"c2257408-632e-4e27-ae69-99046b12b405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3556298518},{"type":"v1","addr":"172.21.15.6:6811","nonce":3556298518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3556298518},{"type":"v1","addr":"172.21.15.6:6813","nonce":3556298518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":3556298518},{"type":"v1","addr":"172.21.15.6:6817","nonce":3556298518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":3556298518},{"type":"v1","addr":"172.21.15.6:6815","nonce":3556298518}]},"public_addr":"172.21.15.6:6811/3556298518","cluster_addr":"172.21.15.6:6813/3556298518","heartbeat_back_addr":"172.21.15.6:6817/3556298518","heartbeat_front_addr":"172.21.15.6:6815/3556298518","state":["exists","up"]},{"osd":2,"uuid":"e56e4fc0-614d-48e6-a2d6-758c34487540","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3201508057},{"type":"v1","addr":"172.21.15.6:6819","nonce":3201508057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3201508057},{"type":"v1","addr":"172.21.15.6:6821","nonce":3201508057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3201508057},{"type":"v1","addr":"172.21.15.6:6825","nonce":3201508057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3201508057},{"type":"v1","addr":"172.21.15.6:6823","nonce":3201508057}]},"public_addr":"172.21.15.6:6819/3201508057","cluster_addr":"172.21.15.6:6821/3201508057","heartbeat_back_addr":"172.21.15.6:6825/3201508057","heartbeat_front_addr":"172.21.15.6:6823/3201508057","state":["exists","up"]},{"osd":3,"uuid":"0b2a19ac-de1e-46f4-a3e0-5a8321672e4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":1692709589},{"type":"v1","addr":"172.21.15.90:6801","nonce":1692709589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1692709589},{"type":"v1","addr":"172.21.15.90:6803","nonce":1692709589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":1692709589},{"type":"v1","addr":"172.21.15.90:6807","nonce":1692709589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":1692709589},{"type":"v1","addr":"172.21.15.90:6805","nonce":1692709589}]},"public_addr":"172.21.15.90:6801/1692709589","cluster_addr":"172.21.15.90:6803/1692709589","heartbeat_back_addr":"172.21.15.90:6807/1692709589","heartbeat_front_addr":"172.21.15.90:6805/1692709589","state":["exists","up"]},{"osd":4,"uuid":"d79e2bb6-cebd-4e3c-8cec-5fac286838e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":204678191},{"type":"v1","addr":"172.21.15.90:6809","nonce":204678191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":204678191},{"type":"v1","addr":"172.21.15.90:6811","nonce":204678191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":204678191},{"type":"v1","addr":"172.21.15.90:6815","nonce":204678191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":204678191},{"type":"v1","addr":"172.21.15.90:6813","nonce":204678191}]},"public_addr":"172.21.15.90:6809/204678191","cluster_addr":"172.21.15.90:6811/204678191","heartbeat_back_addr":"172.21.15.90:6815/204678191","heartbeat_front_addr":"172.21.15.90:6813/204678191","state":["exists","up"]},{"osd":5,"uuid":"155a408e-ef3b-4c0a-8477-47fc6250b589","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3296441764},{"type":"v1","addr":"172.21.15.90:6817","nonce":3296441764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":3296441764},{"type":"v1","addr":"172.21.15.90:6819","nonce":3296441764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":3296441764},{"type":"v1","addr":"172.21.15.90:6823","nonce":3296441764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":3296441764},{"type":"v1","addr":"172.21.15.90:6821","nonce":3296441764}]},"public_addr":"172.21.15.90:6817/3296441764","cluster_addr":"172.21.15.90:6819/3296441764","heartbeat_back_addr":"172.21.15.90:6823/3296441764","heartbeat_front_addr":"172.21.15.90:6821/3296441764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T21:53:10.093731+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T21:53:39.528761+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T21:54:06.766402+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T21:54:30.531184+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T21:54:55.632482+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T21:55:21.026419+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:0/1297965797":"2024-02-10T21:51:56.813980+0000","172.21.15.6:0/2706037691":"2024-02-10T21:51:56.813980+0000","172.21.15.6:0/3672447393":"2024-02-10T21:51:56.813980+0000","172.21.15.6:6801/3321564517":"2024-02-10T21:51:56.813980+0000","172.21.15.6:0/1651082446":"2024-02-10T21:50:40.245871+0000","172.21.15.6:6800/1422219104":"2024-02-10T21:50:40.245871+0000","172.21.15.6:0/3369982019":"2024-02-10T21:50:07.987703+0000","172.21.15.6:6801/2325806844":"2024-02-10T21:50:07.987703+0000","172.21.15.6:6800/3321564517":"2024-02-10T21:51:56.813980+0000","172.21.15.6:0/147879205":"2024-02-10T21:50:07.987703+0000","172.21.15.6:6801/1422219104":"2024-02-10T21:50:40.245871+0000","172.21.15.6:0/2336225467":"2024-02-10T21:50:07.987703+0000","172.21.15.6:6800/2325806844":"2024-02-10T21:50:07.987703+0000","172.21.15.6:0/965195423":"2024-02-10T21:50:40.245871+0000","172.21.15.6:0/362750223":"2024-02-10T21:50:40.245871+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-09T21:57:07.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.335+0000 7f2b32258640 1 -- 172.21.15.6:0/2683967614 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f2b140692e0 msgr2=0x7f2b1406b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:07.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.335+0000 7f2b32258640 1 --2- 172.21.15.6:0/2683967614 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f2b140692e0 0x7f2b1406b790 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2b2002bda0 tx=0x7f2b20009da0 comp rx=0 tx=0).stop 2024-02-09T21:57:07.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.335+0000 7f2b32258640 1 -- 172.21.15.6:0/2683967614 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b2c0fb180 msgr2=0x7f2b2c10ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:07.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.335+0000 7f2b32258640 1 --2- 172.21.15.6:0/2683967614 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b2c0fb180 0x7f2b2c10ba00 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c00eb70 tx=0x7f2b1c00b040 comp rx=0 tx=0).stop 2024-02-09T21:57:07.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.335+0000 7f2b32258640 1 -- 172.21.15.6:0/2683967614 shutdown_connections 2024-02-09T21:57:07.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.335+0000 7f2b32258640 1 --2- 172.21.15.6:0/2683967614 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f2b140692e0 0x7f2b1406b790 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:07.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.335+0000 7f2b32258640 1 --2- 172.21.15.6:0/2683967614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b2c0fc450 0x7f2b2c10bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:07.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.335+0000 7f2b32258640 1 --2- 172.21.15.6:0/2683967614 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b2c0fb180 0x7f2b2c10ba00 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:07.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.336+0000 7f2b32258640 1 -- 172.21.15.6:0/2683967614 >> 172.21.15.6:0/2683967614 conn(0x7f2b2c0f6940 msgr2=0x7f2b2c0ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:57:07.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.336+0000 7f2b32258640 1 -- 172.21.15.6:0/2683967614 shutdown_connections 2024-02-09T21:57:07.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.336+0000 7f2b32258640 1 -- 172.21.15.6:0/2683967614 wait complete. 2024-02-09T21:57:07.352 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-09T21:54:09.146714+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-02-09T21:56:37.425420+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-02-09T21:56:37.727029+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-09T21:57:07.352 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-09T21:57:07.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.464+0000 7ff10411d640 1 -- 172.21.15.6:0/1876771935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0fc0ff750 msgr2=0x7ff0fc101b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:07.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.464+0000 7ff10411d640 1 --2- 172.21.15.6:0/1876771935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0fc0ff750 0x7ff0fc101b30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec0048d0 tx=0x7ff0ec009da0 comp rx=0 tx=0).stop 2024-02-09T21:57:07.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.465+0000 7ff10411d640 1 -- 172.21.15.6:0/1876771935 shutdown_connections 2024-02-09T21:57:07.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.465+0000 7ff10411d640 1 --2- 172.21.15.6:0/1876771935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0fc0ff750 0x7ff0fc101b30 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:07.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.465+0000 7ff10411d640 1 --2- 172.21.15.6:0/1876771935 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0fc0fce30 0x7ff0fc0ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:07.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.465+0000 7ff10411d640 1 -- 172.21.15.6:0/1876771935 >> 172.21.15.6:0/1876771935 conn(0x7ff0fc0f68d0 msgr2=0x7ff0fc0f8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:57:07.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.465+0000 7ff10411d640 1 -- 172.21.15.6:0/1876771935 shutdown_connections 2024-02-09T21:57:07.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.465+0000 7ff10411d640 1 -- 172.21.15.6:0/1876771935 wait complete. 2024-02-09T21:57:07.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.465+0000 7ff10411d640 1 Processor -- start 2024-02-09T21:57:07.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.465+0000 7ff10411d640 1 -- start start 2024-02-09T21:57:07.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.465+0000 7ff10411d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0fc0fce30 0x7ff0fc0fc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:07.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.465+0000 7ff10411d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0fc0ff750 0x7ff0fc0fad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:07.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.465+0000 7ff10411d640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0fc0fb280 con 0x7ff0fc0fce30 2024-02-09T21:57:07.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.465+0000 7ff10411d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0fc0fb3f0 con 0x7ff0fc0ff750 2024-02-09T21:57:07.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.465+0000 7ff101eb9640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0fc0fce30 0x7ff0fc0fc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:07.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.465+0000 7ff101eb9640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0fc0fce30 0x7ff0fc0fc6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57488/0 (socket says 172.21.15.6:57488) 2024-02-09T21:57:07.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.465+0000 7ff101eb9640 1 -- 172.21.15.6:0/211825705 learned_addr learned my addr 172.21.15.6:0/211825705 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:57:07.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.466+0000 7ff1016b8640 1 --2- 172.21.15.6:0/211825705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0fc0ff750 0x7ff0fc0fad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:07.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.466+0000 7ff101eb9640 1 -- 172.21.15.6:0/211825705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0fc0ff750 msgr2=0x7ff0fc0fad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:07.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.466+0000 7ff101eb9640 1 --2- 172.21.15.6:0/211825705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0fc0ff750 0x7ff0fc0fad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:07.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.466+0000 7ff101eb9640 1 -- 172.21.15.6:0/211825705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0ec005040 con 0x7ff0fc0fce30 2024-02-09T21:57:07.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.466+0000 7ff101eb9640 1 --2- 172.21.15.6:0/211825705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0fc0fce30 0x7ff0fc0fc6c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800eb70 tx=0x7ff0f800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:57:07.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.466+0000 7ff0f27fc640 1 -- 172.21.15.6:0/211825705 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0f8007c30 con 0x7ff0fc0fce30 2024-02-09T21:57:07.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.466+0000 7ff0f27fc640 1 -- 172.21.15.6:0/211825705 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff0f800f040 con 0x7ff0fc0fce30 2024-02-09T21:57:07.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.466+0000 7ff10411d640 1 -- 172.21.15.6:0/211825705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0fc0fb670 con 0x7ff0fc0fce30 2024-02-09T21:57:07.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.466+0000 7ff0f27fc640 1 -- 172.21.15.6:0/211825705 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0f80095b0 con 0x7ff0fc0fce30 2024-02-09T21:57:07.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.466+0000 7ff10411d640 1 -- 172.21.15.6:0/211825705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0fc0fbb60 con 0x7ff0fc0fce30 2024-02-09T21:57:07.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.467+0000 7ff10411d640 1 -- 172.21.15.6:0/211825705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0fc064030 con 0x7ff0fc0fce30 2024-02-09T21:57:07.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.468+0000 7ff0f27fc640 1 -- 172.21.15.6:0/211825705 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7ff0f8009730 con 0x7ff0fc0fce30 2024-02-09T21:57:07.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.468+0000 7ff0f27fc640 1 --2- 172.21.15.6:0/211825705 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7ff0e8069210 0x7ff0e806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:07.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.468+0000 7ff0f27fc640 1 -- 172.21.15.6:0/211825705 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7ff0f8086eb0 con 0x7ff0fc0fce30 2024-02-09T21:57:07.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.468+0000 7ff1016b8640 1 --2- 172.21.15.6:0/211825705 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7ff0e8069210 0x7ff0e806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:07.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.469+0000 7ff1016b8640 1 --2- 172.21.15.6:0/211825705 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7ff0e8069210 0x7ff0e806b6c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec000c00 tx=0x7ff0ec009da0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:57:07.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.473+0000 7ff0f27fc640 1 -- 172.21.15.6:0/211825705 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff0f8053210 con 0x7ff0fc0fce30 2024-02-09T21:57:07.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.647+0000 7ff10411d640 1 -- 172.21.15.6:0/211825705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff0fc02a9b0 con 0x7ff0fc0fce30 2024-02-09T21:57:07.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.648+0000 7ff0f27fc640 1 -- 172.21.15.6:0/211825705 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v45) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff0f8056830 con 0x7ff0fc0fce30 2024-02-09T21:57:07.651 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-02-09T21:57:07.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.649+0000 7ff10411d640 1 -- 172.21.15.6:0/211825705 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7ff0e8069210 msgr2=0x7ff0e806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:07.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.649+0000 7ff10411d640 1 --2- 172.21.15.6:0/211825705 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7ff0e8069210 0x7ff0e806b6c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec000c00 tx=0x7ff0ec009da0 comp rx=0 tx=0).stop 2024-02-09T21:57:07.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.649+0000 7ff10411d640 1 -- 172.21.15.6:0/211825705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0fc0fce30 msgr2=0x7ff0fc0fc6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:07.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.649+0000 7ff10411d640 1 --2- 172.21.15.6:0/211825705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0fc0fce30 0x7ff0fc0fc6c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800eb70 tx=0x7ff0f800b040 comp rx=0 tx=0).stop 2024-02-09T21:57:07.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.649+0000 7ff10411d640 1 -- 172.21.15.6:0/211825705 shutdown_connections 2024-02-09T21:57:07.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.649+0000 7ff10411d640 1 --2- 172.21.15.6:0/211825705 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7ff0e8069210 0x7ff0e806b6c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:07.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.649+0000 7ff10411d640 1 --2- 172.21.15.6:0/211825705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0fc0ff750 0x7ff0fc0fad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:07.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.649+0000 7ff10411d640 1 --2- 172.21.15.6:0/211825705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff0fc0fce30 0x7ff0fc0fc6c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:07.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.649+0000 7ff10411d640 1 -- 172.21.15.6:0/211825705 >> 172.21.15.6:0/211825705 conn(0x7ff0fc0f68d0 msgr2=0x7ff0fc0f8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:57:07.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.649+0000 7ff10411d640 1 -- 172.21.15.6:0/211825705 shutdown_connections 2024-02-09T21:57:07.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.650+0000 7ff10411d640 1 -- 172.21.15.6:0/211825705 wait complete. 2024-02-09T21:57:07.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:07 smithi006 ceph-mon[29258]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-09T21:57:07.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:07 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/2683967614' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T21:57:07.665 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-02-09T21:57:07.704 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:07 smithi090 ceph-mon[36996]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-09T21:57:07.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:07 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/2683967614' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T21:57:07.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.762+0000 7f7e5fb9d640 1 -- 172.21.15.6:0/883669821 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e580fc400 msgr2=0x7f7e580fc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:07.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.762+0000 7f7e5fb9d640 1 --2- 172.21.15.6:0/883669821 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e580fc400 0x7f7e580fc870 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7e480048d0 tx=0x7f7e48009da0 comp rx=0 tx=0).stop 2024-02-09T21:57:07.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.762+0000 7f7e5fb9d640 1 -- 172.21.15.6:0/883669821 shutdown_connections 2024-02-09T21:57:07.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.762+0000 7f7e5fb9d640 1 --2- 172.21.15.6:0/883669821 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e580fc400 0x7f7e580fc870 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:07.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.762+0000 7f7e5fb9d640 1 --2- 172.21.15.6:0/883669821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7e580fb1d0 0x7f7e580fb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:07.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.762+0000 7f7e5fb9d640 1 -- 172.21.15.6:0/883669821 >> 172.21.15.6:0/883669821 conn(0x7f7e580f68d0 msgr2=0x7f7e580f8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:57:07.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.762+0000 7f7e5fb9d640 1 -- 172.21.15.6:0/883669821 shutdown_connections 2024-02-09T21:57:07.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.762+0000 7f7e5fb9d640 1 -- 172.21.15.6:0/883669821 wait complete. 2024-02-09T21:57:07.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.763+0000 7f7e5fb9d640 1 Processor -- start 2024-02-09T21:57:07.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.763+0000 7f7e5fb9d640 1 -- start start 2024-02-09T21:57:07.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.763+0000 7f7e5fb9d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e580fb1d0 0x7f7e58064f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:07.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.763+0000 7f7e5fb9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7e580fc400 0x7f7e581119b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:07.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.763+0000 7f7e5fb9d640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e58111ef0 con 0x7f7e580fb1d0 2024-02-09T21:57:07.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.763+0000 7f7e5fb9d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e58112030 con 0x7f7e580fc400 2024-02-09T21:57:07.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.763+0000 7f7e5d939640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e580fb1d0 0x7f7e58064f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:07.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.763+0000 7f7e5d939640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e580fb1d0 0x7f7e58064f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43084/0 (socket says 172.21.15.6:43084) 2024-02-09T21:57:07.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.763+0000 7f7e5d939640 1 -- 172.21.15.6:0/4146026488 learned_addr learned my addr 172.21.15.6:0/4146026488 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:57:07.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.763+0000 7f7e5d138640 1 --2- 172.21.15.6:0/4146026488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7e580fc400 0x7f7e581119b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:07.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.763+0000 7f7e5d939640 1 -- 172.21.15.6:0/4146026488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7e580fc400 msgr2=0x7f7e581119b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:07.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.763+0000 7f7e5d939640 1 --2- 172.21.15.6:0/4146026488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7e580fc400 0x7f7e581119b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:07.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.763+0000 7f7e5d939640 1 -- 172.21.15.6:0/4146026488 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e48005040 con 0x7f7e580fb1d0 2024-02-09T21:57:07.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.763+0000 7f7e5d939640 1 --2- 172.21.15.6:0/4146026488 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e580fb1d0 0x7f7e58064f10 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7e54002a80 tx=0x7f7e54004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:57:07.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.764+0000 7f7e4e7fc640 1 -- 172.21.15.6:0/4146026488 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e54007c30 con 0x7f7e580fb1d0 2024-02-09T21:57:07.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.764+0000 7f7e4e7fc640 1 -- 172.21.15.6:0/4146026488 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7e5400f040 con 0x7f7e580fb1d0 2024-02-09T21:57:07.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.764+0000 7f7e5fb9d640 1 -- 172.21.15.6:0/4146026488 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e581122c0 con 0x7f7e580fb1d0 2024-02-09T21:57:07.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.764+0000 7f7e4e7fc640 1 -- 172.21.15.6:0/4146026488 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e54009650 con 0x7f7e580fb1d0 2024-02-09T21:57:07.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.764+0000 7f7e5fb9d640 1 -- 172.21.15.6:0/4146026488 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e581127e0 con 0x7f7e580fb1d0 2024-02-09T21:57:07.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.765+0000 7f7e5fb9d640 1 -- 172.21.15.6:0/4146026488 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e58103d20 con 0x7f7e580fb1d0 2024-02-09T21:57:07.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.765+0000 7f7e4e7fc640 1 -- 172.21.15.6:0/4146026488 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f7e5400d6b0 con 0x7f7e580fb1d0 2024-02-09T21:57:07.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.765+0000 7f7e4e7fc640 1 --2- 172.21.15.6:0/4146026488 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f7e44069210 0x7f7e4406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:07.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.766+0000 7f7e5d138640 1 --2- 172.21.15.6:0/4146026488 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f7e44069210 0x7f7e4406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:07.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.766+0000 7f7e4e7fc640 1 -- 172.21.15.6:0/4146026488 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f7e54087180 con 0x7f7e580fb1d0 2024-02-09T21:57:07.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.766+0000 7f7e5d138640 1 --2- 172.21.15.6:0/4146026488 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f7e44069210 0x7f7e4406b6c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7e48003080 tx=0x7f7e48009da0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:57:07.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.770+0000 7f7e4e7fc640 1 -- 172.21.15.6:0/4146026488 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7e54053430 con 0x7f7e580fb1d0 2024-02-09T21:57:07.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.938+0000 7f7e5fb9d640 1 -- 172.21.15.6:0/4146026488 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f7e58112cb0 con 0x7f7e580fb1d0 2024-02-09T21:57:07.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.939+0000 7f7e4e7fc640 1 -- 172.21.15.6:0/4146026488 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v45) v1 ==== 107+0+11 (secure 0 0 0) 0x7f7e54056a50 con 0x7f7e580fb1d0 2024-02-09T21:57:07.944 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-02-09T21:57:07.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.941+0000 7f7e5fb9d640 1 -- 172.21.15.6:0/4146026488 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f7e44069210 msgr2=0x7f7e4406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:07.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.941+0000 7f7e5fb9d640 1 --2- 172.21.15.6:0/4146026488 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f7e44069210 0x7f7e4406b6c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7e48003080 tx=0x7f7e48009da0 comp rx=0 tx=0).stop 2024-02-09T21:57:07.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.941+0000 7f7e5fb9d640 1 -- 172.21.15.6:0/4146026488 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e580fb1d0 msgr2=0x7f7e58064f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:07.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.941+0000 7f7e5fb9d640 1 --2- 172.21.15.6:0/4146026488 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e580fb1d0 0x7f7e58064f10 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7e54002a80 tx=0x7f7e54004570 comp rx=0 tx=0).stop 2024-02-09T21:57:07.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.941+0000 7f7e5fb9d640 1 -- 172.21.15.6:0/4146026488 shutdown_connections 2024-02-09T21:57:07.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.941+0000 7f7e5fb9d640 1 --2- 172.21.15.6:0/4146026488 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f7e44069210 0x7f7e4406b6c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:07.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.941+0000 7f7e5fb9d640 1 --2- 172.21.15.6:0/4146026488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7e580fc400 0x7f7e581119b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:07.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.941+0000 7f7e5fb9d640 1 --2- 172.21.15.6:0/4146026488 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e580fb1d0 0x7f7e58064f10 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:07.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.941+0000 7f7e5fb9d640 1 -- 172.21.15.6:0/4146026488 >> 172.21.15.6:0/4146026488 conn(0x7f7e580f68d0 msgr2=0x7f7e580ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:57:07.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.941+0000 7f7e5fb9d640 1 -- 172.21.15.6:0/4146026488 shutdown_connections 2024-02-09T21:57:07.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:07.941+0000 7f7e5fb9d640 1 -- 172.21.15.6:0/4146026488 wait complete. 2024-02-09T21:57:07.956 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-02-09T21:57:08.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.058+0000 7fe48691d640 1 -- 172.21.15.6:0/206057213 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe4800fa5d0 msgr2=0x7fe4800fa9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:08.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.058+0000 7fe48691d640 1 --2- 172.21.15.6:0/206057213 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe4800fa5d0 0x7fe4800fa9e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe4700048d0 tx=0x7fe470009da0 comp rx=0 tx=0).stop 2024-02-09T21:57:08.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.058+0000 7fe48691d640 1 -- 172.21.15.6:0/206057213 shutdown_connections 2024-02-09T21:57:08.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.058+0000 7fe48691d640 1 --2- 172.21.15.6:0/206057213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4800fd5e0 0x7fe4800fda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:08.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.058+0000 7fe48691d640 1 --2- 172.21.15.6:0/206057213 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe4800fa5d0 0x7fe4800fa9e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:08.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.058+0000 7fe48691d640 1 -- 172.21.15.6:0/206057213 >> 172.21.15.6:0/206057213 conn(0x7fe4800f5ff0 msgr2=0x7fe4800f8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:57:08.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.058+0000 7fe48691d640 1 -- 172.21.15.6:0/206057213 shutdown_connections 2024-02-09T21:57:08.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.058+0000 7fe48691d640 1 -- 172.21.15.6:0/206057213 wait complete. 2024-02-09T21:57:08.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.058+0000 7fe48691d640 1 Processor -- start 2024-02-09T21:57:08.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.058+0000 7fe48691d640 1 -- start start 2024-02-09T21:57:08.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.058+0000 7fe48691d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe4800fa5d0 0x7fe48010cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:08.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.058+0000 7fe48691d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4800fd5e0 0x7fe48010d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:08.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.058+0000 7fe48691d640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48010d790 con 0x7fe4800fa5d0 2024-02-09T21:57:08.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.058+0000 7fe48691d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4801121a0 con 0x7fe4800fd5e0 2024-02-09T21:57:08.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.059+0000 7fe47ffff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe4800fa5d0 0x7fe48010cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:08.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.059+0000 7fe47ffff640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe4800fa5d0 0x7fe48010cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43090/0 (socket says 172.21.15.6:43090) 2024-02-09T21:57:08.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.059+0000 7fe47ffff640 1 -- 172.21.15.6:0/2188345310 learned_addr learned my addr 172.21.15.6:0/2188345310 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:57:08.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.059+0000 7fe47ffff640 1 -- 172.21.15.6:0/2188345310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4800fd5e0 msgr2=0x7fe48010d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:08.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.059+0000 7fe47ffff640 1 --2- 172.21.15.6:0/2188345310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4800fd5e0 0x7fe48010d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:08.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.059+0000 7fe47ffff640 1 -- 172.21.15.6:0/2188345310 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe470005040 con 0x7fe4800fa5d0 2024-02-09T21:57:08.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.059+0000 7fe47ffff640 1 --2- 172.21.15.6:0/2188345310 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe4800fa5d0 0x7fe48010cc00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe470034040 tx=0x7fe470009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:57:08.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.059+0000 7fe47cff9640 1 -- 172.21.15.6:0/2188345310 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe470009590 con 0x7fe4800fa5d0 2024-02-09T21:57:08.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.059+0000 7fe47cff9640 1 -- 172.21.15.6:0/2188345310 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe4700096f0 con 0x7fe4800fa5d0 2024-02-09T21:57:08.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.059+0000 7fe48691d640 1 -- 172.21.15.6:0/2188345310 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4801122e0 con 0x7fe4800fa5d0 2024-02-09T21:57:08.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.059+0000 7fe47cff9640 1 -- 172.21.15.6:0/2188345310 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe47003b6b0 con 0x7fe4800fa5d0 2024-02-09T21:57:08.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.059+0000 7fe48691d640 1 -- 172.21.15.6:0/2188345310 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4801127b0 con 0x7fe4800fa5d0 2024-02-09T21:57:08.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.060+0000 7fe48691d640 1 -- 172.21.15.6:0/2188345310 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe480104e50 con 0x7fe4800fa5d0 2024-02-09T21:57:08.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.060+0000 7fe47cff9640 1 -- 172.21.15.6:0/2188345310 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fe47002f070 con 0x7fe4800fa5d0 2024-02-09T21:57:08.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.061+0000 7fe47cff9640 1 --2- 172.21.15.6:0/2188345310 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fe468069210 0x7fe46806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:08.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.061+0000 7fe47cff9640 1 -- 172.21.15.6:0/2188345310 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fe4700a9940 con 0x7fe4800fa5d0 2024-02-09T21:57:08.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.061+0000 7fe47f7fe640 1 --2- 172.21.15.6:0/2188345310 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fe468069210 0x7fe46806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:08.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.061+0000 7fe47f7fe640 1 --2- 172.21.15.6:0/2188345310 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fe468069210 0x7fe46806b6c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe480065680 tx=0x7fe474007c50 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:57:08.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.065+0000 7fe47cff9640 1 -- 172.21.15.6:0/2188345310 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe470075bf0 con 0x7fe4800fa5d0 2024-02-09T21:57:08.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.236+0000 7fe48691d640 1 -- 172.21.15.6:0/2188345310 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fe480112c00 con 0x7fe4800fa5d0 2024-02-09T21:57:08.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.236+0000 7fe47cff9640 1 -- 172.21.15.6:0/2188345310 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v45) v1 ==== 107+0+11 (secure 0 0 0) 0x7fe470079210 con 0x7fe4800fa5d0 2024-02-09T21:57:08.241 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-02-09T21:57:08.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.238+0000 7fe48691d640 1 -- 172.21.15.6:0/2188345310 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fe468069210 msgr2=0x7fe46806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:08.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.238+0000 7fe48691d640 1 --2- 172.21.15.6:0/2188345310 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fe468069210 0x7fe46806b6c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe480065680 tx=0x7fe474007c50 comp rx=0 tx=0).stop 2024-02-09T21:57:08.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.238+0000 7fe48691d640 1 -- 172.21.15.6:0/2188345310 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe4800fa5d0 msgr2=0x7fe48010cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:08.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.238+0000 7fe48691d640 1 --2- 172.21.15.6:0/2188345310 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe4800fa5d0 0x7fe48010cc00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe470034040 tx=0x7fe470009da0 comp rx=0 tx=0).stop 2024-02-09T21:57:08.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.238+0000 7fe48691d640 1 -- 172.21.15.6:0/2188345310 shutdown_connections 2024-02-09T21:57:08.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.238+0000 7fe48691d640 1 --2- 172.21.15.6:0/2188345310 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fe468069210 0x7fe46806b6c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:08.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.238+0000 7fe48691d640 1 --2- 172.21.15.6:0/2188345310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4800fd5e0 0x7fe48010d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:08.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.238+0000 7fe48691d640 1 --2- 172.21.15.6:0/2188345310 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe4800fa5d0 0x7fe48010cc00 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:08.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.238+0000 7fe48691d640 1 -- 172.21.15.6:0/2188345310 >> 172.21.15.6:0/2188345310 conn(0x7fe4800f5ff0 msgr2=0x7fe480100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:57:08.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.238+0000 7fe48691d640 1 -- 172.21.15.6:0/2188345310 shutdown_connections 2024-02-09T21:57:08.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.239+0000 7fe48691d640 1 -- 172.21.15.6:0/2188345310 wait complete. 2024-02-09T21:57:08.254 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T21:57:08.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.348+0000 7f6f1bf5d640 1 -- 172.21.15.6:0/2195173302 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f140ff700 msgr2=0x7f6f14101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:08.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.348+0000 7f6f1bf5d640 1 --2- 172.21.15.6:0/2195173302 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f140ff700 0x7f6f14101ae0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6f100048d0 tx=0x7f6f10009da0 comp rx=0 tx=0).stop 2024-02-09T21:57:08.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.348+0000 7f6f1bf5d640 1 -- 172.21.15.6:0/2195173302 shutdown_connections 2024-02-09T21:57:08.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.348+0000 7f6f1bf5d640 1 --2- 172.21.15.6:0/2195173302 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f140ff700 0x7f6f14101ae0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:08.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.348+0000 7f6f1bf5d640 1 --2- 172.21.15.6:0/2195173302 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f140fcde0 0x7f6f140ff1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:08.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.348+0000 7f6f1bf5d640 1 -- 172.21.15.6:0/2195173302 >> 172.21.15.6:0/2195173302 conn(0x7f6f140f6860 msgr2=0x7f6f140f8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:57:08.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.348+0000 7f6f1bf5d640 1 -- 172.21.15.6:0/2195173302 shutdown_connections 2024-02-09T21:57:08.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.349+0000 7f6f1bf5d640 1 -- 172.21.15.6:0/2195173302 wait complete. 2024-02-09T21:57:08.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.349+0000 7f6f1bf5d640 1 Processor -- start 2024-02-09T21:57:08.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.349+0000 7f6f1bf5d640 1 -- start start 2024-02-09T21:57:08.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.349+0000 7f6f1bf5d640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f140fcde0 0x7f6f140fc670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:08.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.349+0000 7f6f1bf5d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f140ff700 0x7f6f140facf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.349+0000 7f6f1bf5d640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f140fb230 con 0x7f6f140fcde0 2024-02-09T21:57:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.349+0000 7f6f1bf5d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f140fb3a0 con 0x7f6f140ff700 2024-02-09T21:57:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.349+0000 7f6f19cf9640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f140fcde0 0x7f6f140fc670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.349+0000 7f6f19cf9640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f140fcde0 0x7f6f140fc670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43104/0 (socket says 172.21.15.6:43104) 2024-02-09T21:57:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.349+0000 7f6f19cf9640 1 -- 172.21.15.6:0/3431706882 learned_addr learned my addr 172.21.15.6:0/3431706882 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:57:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.350+0000 7f6f194f8640 1 --2- 172.21.15.6:0/3431706882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f140ff700 0x7f6f140facf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.350+0000 7f6f19cf9640 1 -- 172.21.15.6:0/3431706882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f140ff700 msgr2=0x7f6f140facf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.350+0000 7f6f19cf9640 1 --2- 172.21.15.6:0/3431706882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f140ff700 0x7f6f140facf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.350+0000 7f6f19cf9640 1 -- 172.21.15.6:0/3431706882 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f10005040 con 0x7f6f140fcde0 2024-02-09T21:57:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.350+0000 7f6f194f8640 1 --2- 172.21.15.6:0/3431706882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f140ff700 0x7f6f140facf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T21:57:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.350+0000 7f6f19cf9640 1 --2- 172.21.15.6:0/3431706882 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f140fcde0 0x7f6f140fc670 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6f0800c790 tx=0x7f6f080043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:57:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.350+0000 7f6f067fc640 1 -- 172.21.15.6:0/3431706882 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f0800a820 con 0x7f6f140fcde0 2024-02-09T21:57:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.350+0000 7f6f067fc640 1 -- 172.21.15.6:0/3431706882 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6f0800ae50 con 0x7f6f140fcde0 2024-02-09T21:57:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.350+0000 7f6f1bf5d640 1 -- 172.21.15.6:0/3431706882 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f140fb680 con 0x7f6f140fcde0 2024-02-09T21:57:08.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.350+0000 7f6f067fc640 1 -- 172.21.15.6:0/3431706882 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f080115e0 con 0x7f6f140fcde0 2024-02-09T21:57:08.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.350+0000 7f6f1bf5d640 1 -- 172.21.15.6:0/3431706882 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f14114ca0 con 0x7f6f140fcde0 2024-02-09T21:57:08.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.351+0000 7f6f1bf5d640 1 -- 172.21.15.6:0/3431706882 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f14104de0 con 0x7f6f140fcde0 2024-02-09T21:57:08.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.352+0000 7f6f067fc640 1 -- 172.21.15.6:0/3431706882 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f6f08012400 con 0x7f6f140fcde0 2024-02-09T21:57:08.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.352+0000 7f6f067fc640 1 --2- 172.21.15.6:0/3431706882 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f6f00069210 0x7f6f0006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:08.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.352+0000 7f6f067fc640 1 -- 172.21.15.6:0/3431706882 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f6f08087350 con 0x7f6f140fcde0 2024-02-09T21:57:08.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.352+0000 7f6f194f8640 1 --2- 172.21.15.6:0/3431706882 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f6f00069210 0x7f6f0006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:08.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.352+0000 7f6f194f8640 1 --2- 172.21.15.6:0/3431706882 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f6f00069210 0x7f6f0006b6c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6f1002bda0 tx=0x7f6f10009da0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:57:08.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.358+0000 7f6f067fc640 1 -- 172.21.15.6:0/3431706882 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6f08053680 con 0x7f6f140fcde0 2024-02-09T21:57:08.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.582+0000 7f6f1bf5d640 1 -- 172.21.15.6:0/3431706882 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f14114e30 con 0x7f6f140fcde0 2024-02-09T21:57:08.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.583+0000 7f6f067fc640 1 -- 172.21.15.6:0/3431706882 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4708 (secure 0 0 0) 0x7f6f08056ca0 con 0x7f6f140fcde0 2024-02-09T21:57:08.588 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-02-09T21:57:08.588 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":18,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T21:56:38.486224+0000","modified":"2024-02-09T21:56:55.295828+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14458,"mds_1":24271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14458":{"gid":14458,"name":"cephfs.smithi006.ypgiqs","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6827/669315024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":669315024},{"type":"v1","addr":"172.21.15.6:6827","nonce":669315024}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14472":{"gid":14472,"name":"cephfs.smithi006.qyamnu","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.6:6829/3025444869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3025444869},{"type":"v1","addr":"172.21.15.6:6829","nonce":3025444869}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24271":{"gid":24271,"name":"cephfs.smithi090.cgsuda","rank":1,"incarnation":9,"state":"up:active","state_seq":4,"addr":"172.21.15.90:6825/2739488453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2739488453},{"type":"v1","addr":"172.21.15.90:6825","nonce":2739488453}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24287":{"gid":24287,"name":"cephfs.smithi090.mipeaa","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.90:6827/1908638886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1908638886},{"type":"v1","addr":"172.21.15.90:6827","nonce":1908638886}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2024-02-09T21:57:08.588 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 18 2024-02-09T21:57:08.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.584+0000 7f6f1bf5d640 1 -- 172.21.15.6:0/3431706882 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f6f00069210 msgr2=0x7f6f0006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:08.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.584+0000 7f6f1bf5d640 1 --2- 172.21.15.6:0/3431706882 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f6f00069210 0x7f6f0006b6c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6f1002bda0 tx=0x7f6f10009da0 comp rx=0 tx=0).stop 2024-02-09T21:57:08.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.585+0000 7f6f1bf5d640 1 -- 172.21.15.6:0/3431706882 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f140fcde0 msgr2=0x7f6f140fc670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:08.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.585+0000 7f6f1bf5d640 1 --2- 172.21.15.6:0/3431706882 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f140fcde0 0x7f6f140fc670 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6f0800c790 tx=0x7f6f080043c0 comp rx=0 tx=0).stop 2024-02-09T21:57:08.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.585+0000 7f6f1bf5d640 1 -- 172.21.15.6:0/3431706882 shutdown_connections 2024-02-09T21:57:08.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.585+0000 7f6f1bf5d640 1 --2- 172.21.15.6:0/3431706882 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f6f00069210 0x7f6f0006b6c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:08.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.585+0000 7f6f1bf5d640 1 --2- 172.21.15.6:0/3431706882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f140ff700 0x7f6f140facf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:08.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.585+0000 7f6f1bf5d640 1 --2- 172.21.15.6:0/3431706882 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f140fcde0 0x7f6f140fc670 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:08.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.585+0000 7f6f1bf5d640 1 -- 172.21.15.6:0/3431706882 >> 172.21.15.6:0/3431706882 conn(0x7f6f140f6860 msgr2=0x7f6f140f8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:57:08.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.585+0000 7f6f1bf5d640 1 -- 172.21.15.6:0/3431706882 shutdown_connections 2024-02-09T21:57:08.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:08.585+0000 7f6f1bf5d640 1 -- 172.21.15.6:0/3431706882 wait complete. 2024-02-09T21:57:08.600 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 18, 'max_mds': 2, 'flags': 50} 2024-02-09T21:57:08.600 INFO:teuthology.run_tasks:Running task kclient... 2024-02-09T21:57:08.627 INFO:tasks.kclient:Mounting kernel clients... 2024-02-09T21:57:08.627 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-02-09T21:57:08.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:08.627 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-02-09T21:57:08.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:08.643 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link delete ceph-brx 2024-02-09T21:57:08.711 INFO:teuthology.orchestra.run.smithi006.stderr:Cannot find device "ceph-brx" 2024-02-09T21:57:08.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T21:57:08.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:08.713 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-02-09T21:57:08.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:08.732 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link delete ceph-brx 2024-02-09T21:57:08.809 INFO:teuthology.orchestra.run.smithi090.stderr:Cannot find device "ceph-brx" 2024-02-09T21:57:08.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T21:57:08.810 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-02-09T21:57:08.811 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-09T21:57:08.811 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-09T21:57:08.811 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph fs ls 2024-02-09T21:57:08.834 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:08 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:57:08.835 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:08 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:57:08.835 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:08 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:57:08.835 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:08 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:57:08.835 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:08 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:57:08.835 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:08 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/211825705' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T21:57:08.835 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:08 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/4146026488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-02-09T21:57:08.835 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:08 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/2188345310' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-02-09T21:57:08.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:08 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:57:08.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:08 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:57:08.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:08 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:57:08.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:08 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:57:08.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:08 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:57:08.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:08 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/211825705' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T21:57:08.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:08 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/4146026488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-02-09T21:57:08.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:08 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/2188345310' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-02-09T21:57:09.071 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:57:09.875 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:09 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/3431706882' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T21:57:09.876 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:09 smithi006 ceph-mon[29258]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-09T21:57:09.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:09 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/3431706882' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T21:57:09.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:09 smithi090 ceph-mon[36996]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-09T21:57:10.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.674+0000 7f33455c2700 1 -- 172.21.15.6:0/4258866749 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3340102880 msgr2=0x7f33400fc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:10.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.674+0000 7f33455c2700 1 --2- 172.21.15.6:0/4258866749 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3340102880 0x7f33400fc680 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3328004940 tx=0x7f3328009e30 comp rx=0 tx=0).stop 2024-02-09T21:57:10.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.674+0000 7f33455c2700 1 -- 172.21.15.6:0/4258866749 shutdown_connections 2024-02-09T21:57:10.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.674+0000 7f33455c2700 1 --2- 172.21.15.6:0/4258866749 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3340102880 0x7f33400fc680 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:10.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.674+0000 7f33455c2700 1 --2- 172.21.15.6:0/4258866749 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3340101ee0 0x7f33401022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:10.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.674+0000 7f33455c2700 1 -- 172.21.15.6:0/4258866749 >> 172.21.15.6:0/4258866749 conn(0x7f3340071560 msgr2=0x7f3340071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:57:10.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.674+0000 7f33455c2700 1 -- 172.21.15.6:0/4258866749 shutdown_connections 2024-02-09T21:57:10.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.674+0000 7f33455c2700 1 -- 172.21.15.6:0/4258866749 wait complete. 2024-02-09T21:57:10.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.674+0000 7f33455c2700 1 Processor -- start 2024-02-09T21:57:10.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.674+0000 7f33455c2700 1 -- start start 2024-02-09T21:57:10.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.675+0000 7f33455c2700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3340101ee0 0x7f334010eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:10.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.675+0000 7f33455c2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3340102880 0x7f334010f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:10.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.675+0000 7f33455c2700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334010f640 con 0x7f3340101ee0 2024-02-09T21:57:10.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.675+0000 7f33455c2700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334010f7b0 con 0x7f3340102880 2024-02-09T21:57:10.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.675+0000 7f333effd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3340101ee0 0x7f334010eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:10.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.675+0000 7f333effd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3340101ee0 0x7f334010eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43126/0 (socket says 172.21.15.6:43126) 2024-02-09T21:57:10.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.675+0000 7f333effd700 1 -- 172.21.15.6:0/1567188201 learned_addr learned my addr 172.21.15.6:0/1567188201 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:57:10.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.675+0000 7f333e7fc700 1 --2- 172.21.15.6:0/1567188201 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3340102880 0x7f334010f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:10.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.675+0000 7f333effd700 1 -- 172.21.15.6:0/1567188201 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3340102880 msgr2=0x7f334010f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:10.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.675+0000 7f333effd700 1 --2- 172.21.15.6:0/1567188201 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3340102880 0x7f334010f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:10.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.675+0000 7f333effd700 1 -- 172.21.15.6:0/1567188201 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3328005040 con 0x7f3340101ee0 2024-02-09T21:57:10.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.675+0000 7f333e7fc700 1 --2- 172.21.15.6:0/1567188201 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3340102880 0x7f334010f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T21:57:10.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.675+0000 7f333effd700 1 --2- 172.21.15.6:0/1567188201 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3340101ee0 0x7f334010eb70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3338002a50 tx=0x7f33380046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:57:10.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.675+0000 7f33377fe700 1 -- 172.21.15.6:0/1567188201 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3338007cc0 con 0x7f3340101ee0 2024-02-09T21:57:10.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.675+0000 7f33455c2700 1 -- 172.21.15.6:0/1567188201 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3340117b40 con 0x7f3340101ee0 2024-02-09T21:57:10.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.675+0000 7f33377fe700 1 -- 172.21.15.6:0/1567188201 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f333800f040 con 0x7f3340101ee0 2024-02-09T21:57:10.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.675+0000 7f33377fe700 1 -- 172.21.15.6:0/1567188201 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3338009640 con 0x7f3340101ee0 2024-02-09T21:57:10.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.675+0000 7f33455c2700 1 -- 172.21.15.6:0/1567188201 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3340118000 con 0x7f3340101ee0 2024-02-09T21:57:10.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.676+0000 7f33455c2700 1 -- 172.21.15.6:0/1567188201 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33401043b0 con 0x7f3340101ee0 2024-02-09T21:57:10.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.677+0000 7f33377fe700 1 -- 172.21.15.6:0/1567188201 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f33380097a0 con 0x7f3340101ee0 2024-02-09T21:57:10.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.677+0000 7f33377fe700 1 --2- 172.21.15.6:0/1567188201 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f332c0692a0 0x7f332c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:10.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.677+0000 7f33377fe700 1 -- 172.21.15.6:0/1567188201 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f3338086fc0 con 0x7f3340101ee0 2024-02-09T21:57:10.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.677+0000 7f333e7fc700 1 --2- 172.21.15.6:0/1567188201 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f332c0692a0 0x7f332c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:10.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.678+0000 7f333e7fc700 1 --2- 172.21.15.6:0/1567188201 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f332c0692a0 0x7f332c06b750 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f33400fc050 tx=0x7f3328009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:57:10.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.681+0000 7f33377fe700 1 -- 172.21.15.6:0/1567188201 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3338053270 con 0x7f3340101ee0 2024-02-09T21:57:10.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.917+0000 7f33455c2700 1 -- 172.21.15.6:0/1567188201 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f33400fbd90 con 0x7f3340101ee0 2024-02-09T21:57:10.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.918+0000 7f33377fe700 1 -- 172.21.15.6:0/1567188201 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) v1 ==== 53+0+83 (secure 0 0 0) 0x7f3338056890 con 0x7f3340101ee0 2024-02-09T21:57:10.922 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-02-09T21:57:10.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.919+0000 7f33455c2700 1 -- 172.21.15.6:0/1567188201 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f332c0692a0 msgr2=0x7f332c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:10.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.919+0000 7f33455c2700 1 --2- 172.21.15.6:0/1567188201 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f332c0692a0 0x7f332c06b750 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f33400fc050 tx=0x7f3328009e30 comp rx=0 tx=0).stop 2024-02-09T21:57:10.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.919+0000 7f33455c2700 1 -- 172.21.15.6:0/1567188201 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3340101ee0 msgr2=0x7f334010eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:10.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.919+0000 7f33455c2700 1 --2- 172.21.15.6:0/1567188201 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3340101ee0 0x7f334010eb70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3338002a50 tx=0x7f33380046b0 comp rx=0 tx=0).stop 2024-02-09T21:57:10.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.920+0000 7f33455c2700 1 -- 172.21.15.6:0/1567188201 shutdown_connections 2024-02-09T21:57:10.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.920+0000 7f33455c2700 1 --2- 172.21.15.6:0/1567188201 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f332c0692a0 0x7f332c06b750 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:10.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.920+0000 7f33455c2700 1 --2- 172.21.15.6:0/1567188201 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3340101ee0 0x7f334010eb70 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:10.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.920+0000 7f33455c2700 1 --2- 172.21.15.6:0/1567188201 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3340102880 0x7f334010f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:10.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.920+0000 7f33455c2700 1 -- 172.21.15.6:0/1567188201 >> 172.21.15.6:0/1567188201 conn(0x7f3340071560 msgr2=0x7f33400fa570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:57:10.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.920+0000 7f33455c2700 1 -- 172.21.15.6:0/1567188201 shutdown_connections 2024-02-09T21:57:10.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:10.920+0000 7f33455c2700 1 -- 172.21.15.6:0/1567188201 wait complete. 2024-02-09T21:57:11.345 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:10 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/1567188201' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-09T21:57:11.428 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:10 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/1567188201' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-09T21:57:11.531 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-02-09T21:57:11.531 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-09T21:57:11.531 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-02-09T21:57:11.532 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-02-09T21:57:11.532 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-09T21:57:11.532 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-02-09T21:57:11.532 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-09T21:57:11.532 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-09T21:57:11.532 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-09T21:57:11.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:11.532 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-02-09T21:57:11.588 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-09T21:57:11.589 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-09T21:57:11.589 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-02-09T21:57:11.589 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-02-09T21:57:11.589 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-02-09T21:57:11.589 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-02-09T21:57:11.589 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-09T21:57:11.589 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-02-09T21:57:11.589 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-02-09T21:57:11.589 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-09T21:57:11.590 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-02-09T21:57:11.590 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-02-09T21:57:11.590 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-09T21:57:11.590 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-02-09T21:57:11.590 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-02-09T21:57:11.590 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-02-09T21:57:11.590 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42231sec preferred_lft 42231sec 2024-02-09T21:57:11.590 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-02-09T21:57:11.590 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-02-09T21:57:11.590 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-09T21:57:11.590 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-02-09T21:57:11.590 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-02-09T21:57:11.591 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-02-09T21:57:11.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:11.591 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-02-09T21:57:11.591 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-02-09T21:57:11.591 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link add name ceph-brx type bridge 2024-02-09T21:57:11.591 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip addr flush dev ceph-brx 2024-02-09T21:57:11.591 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set ceph-brx up 2024-02-09T21:57:11.591 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-02-09T21:57:11.591 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-02-09T21:57:11.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:11.754 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-09T21:57:11.780 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-02-09T21:57:11.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:11.782 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-02-09T21:57:11.856 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-02-09T21:57:11.856 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-09T21:57:11.856 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-02-09T21:57:11.856 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-02-09T21:57:11.856 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-09T21:57:11.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:11.857 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-02-09T21:57:11.857 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-02-09T21:57:11.857 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-02-09T21:57:11.857 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-02-09T21:57:11.857 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-02-09T21:57:11.857 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-02-09T21:57:12.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:12.164 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-02-09T21:57:12.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:12.178 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list-id 2024-02-09T21:57:12.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:12.233 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-02-09T21:57:12.233 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-02-09T21:57:12.233 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-09T21:57:12.233 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-02-09T21:57:12.233 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-02-09T21:57:12.298 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:11 smithi006 ceph-mon[29258]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-09T21:57:12.298 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:11 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:57:12.316 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-02-09T21:57:12.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:12.316 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-02-09T21:57:12.316 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-02-09T21:57:12.316 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-02-09T21:57:12.316 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-02-09T21:57:12.317 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-02-09T21:57:12.317 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-02-09T21:57:12.317 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-02-09T21:57:12.317 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-02-09T21:57:12.454 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:11 smithi090 ceph-mon[36996]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-09T21:57:12.455 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:11 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:57:12.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:12.461 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-02-09T21:57:12.461 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-02-09T21:57:12.461 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-02-09T21:57:12.461 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set dev brx.0 master ceph-brx 2024-02-09T21:57:12.461 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-02-09T21:57:12.502 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-09T21:57:12.502 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-02-09T21:57:12.557 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-02-09T21:57:12.557 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-09T21:57:12.557 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-02-09T21:57:12.611 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-09T21:57:12.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:12.611 DEBUG:teuthology.orchestra.run.smithi006:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-09T21:57:12.697 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-09T21:57:12.697 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-09T21:57:12.697 INFO:teuthology.orchestra.run.smithi006.stdout:invalid new device string format 2024-02-09T21:57:12.810 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-09T21:57:12.810 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-09T21:57:12.810 INFO:teuthology.orchestra.run.smithi006.stdout:invalid new device string format 2024-02-09T21:57:12.811 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.90:3300" 2024-02-09T21:57:12.811 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-09T21:57:12.816 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-09T21:57:12.816 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-09T21:57:12.816 INFO:teuthology.orchestra.run.smithi006.stdout:invalid new device string format 2024-02-09T21:57:12.816 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.90:3300" 2024-02-09T21:57:12.816 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-09T21:57:12.816 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: resolved to: "172.21.15.6:6789,172.21.15.90:6789" 2024-02-09T21:57:12.816 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.6:6789,172.21.15.90:6789:/ 2024-02-09T21:57:12.816 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=ee26d820-c794-11ee-95b8-87774f69a715" will pass to kernel 2024-02-09T21:57:12.819 INFO:teuthology.orchestra.run.smithi006.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-02-09T21:57:12.819 INFO:teuthology.orchestra.run.smithi006.stdout: You just mounted a file system that supports labels which does not 2024-02-09T21:57:12.819 INFO:teuthology.orchestra.run.smithi006.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-02-09T21:57:12.819 INFO:teuthology.orchestra.run.smithi006.stdout: applications will generate AVC messages and not be allowed access to 2024-02-09T21:57:12.819 INFO:teuthology.orchestra.run.smithi006.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-02-09T21:57:12.820 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-09T21:57:12.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:12.821 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-02-09T21:57:12.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:12.848 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-09T21:57:12.908 INFO:teuthology.orchestra.run.smithi006.stdout:client24311 2024-02-09T21:57:12.909 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-02-09T21:57:12.909 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-09T21:57:12.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:12.965 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-09T21:57:13.024 INFO:teuthology.orchestra.run.smithi006.stdout:client24311 2024-02-09T21:57:13.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:13.025 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-09T21:57:13.084 INFO:teuthology.orchestra.run.smithi006.stdout:client24311 2024-02-09T21:57:13.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:13.084 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/ee26d820-c794-11ee-95b8-87774f69a715.client24311/status') 2024-02-09T21:57:13.154 INFO:teuthology.orchestra.run.smithi006.stdout:instance: client.24311 (3)172.21.15.6:0/3137898268 2024-02-09T21:57:13.154 INFO:teuthology.orchestra.run.smithi006.stdout:blocklisted: false 2024-02-09T21:57:13.154 INFO:teuthology.orchestra.run.smithi006.stderr:0+1 records in 2024-02-09T21:57:13.154 INFO:teuthology.orchestra.run.smithi006.stderr:0+1 records out 2024-02-09T21:57:13.154 INFO:teuthology.orchestra.run.smithi006.stderr:70 bytes copied, 2.2022e-05 s, 3.2 MB/s 2024-02-09T21:57:13.156 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-02-09T21:57:13.156 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-09T21:57:13.156 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-02-09T21:57:13.156 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ee26d820-c794-11ee-95b8-87774f69a715 -- ceph fs ls 2024-02-09T21:57:13.456 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:57:14.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:13 smithi006 ceph-mon[29258]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-02-09T21:57:14.454 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:13 smithi090 ceph-mon[36996]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-02-09T21:57:15.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.061+0000 7f8af44e3700 1 -- 172.21.15.6:0/531255961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aec101e40 msgr2=0x7f8aec102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:15.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.061+0000 7f8af44e3700 1 --2- 172.21.15.6:0/531255961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aec101e40 0x7f8aec102210 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8adc004940 tx=0x7f8adc009e30 comp rx=0 tx=0).stop 2024-02-09T21:57:15.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.061+0000 7f8af44e3700 1 -- 172.21.15.6:0/531255961 shutdown_connections 2024-02-09T21:57:15.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.061+0000 7f8af44e3700 1 --2- 172.21.15.6:0/531255961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8aec1027e0 0x7f8aec0fbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:15.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.061+0000 7f8af44e3700 1 --2- 172.21.15.6:0/531255961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aec101e40 0x7f8aec102210 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:15.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.061+0000 7f8af44e3700 1 -- 172.21.15.6:0/531255961 >> 172.21.15.6:0/531255961 conn(0x7f8aec071d90 msgr2=0x7f8aec072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:57:15.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.061+0000 7f8af44e3700 1 -- 172.21.15.6:0/531255961 shutdown_connections 2024-02-09T21:57:15.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.061+0000 7f8af44e3700 1 -- 172.21.15.6:0/531255961 wait complete. 2024-02-09T21:57:15.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.062+0000 7f8af44e3700 1 Processor -- start 2024-02-09T21:57:15.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.062+0000 7f8af44e3700 1 -- start start 2024-02-09T21:57:15.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.062+0000 7f8af44e3700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8aec101e40 0x7f8aec102cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:15.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.062+0000 7f8af44e3700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aec1027e0 0x7f8aec103200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:15.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.062+0000 7f8af44e3700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec103800 con 0x7f8aec1027e0 2024-02-09T21:57:15.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.062+0000 7f8af44e3700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec103970 con 0x7f8aec101e40 2024-02-09T21:57:15.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.062+0000 7f8af1a7e700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aec1027e0 0x7f8aec103200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:15.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.062+0000 7f8af1a7e700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aec1027e0 0x7f8aec103200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43152/0 (socket says 172.21.15.6:43152) 2024-02-09T21:57:15.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.062+0000 7f8af1a7e700 1 -- 172.21.15.6:0/3820350022 learned_addr learned my addr 172.21.15.6:0/3820350022 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:57:15.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.062+0000 7f8af227f700 1 --2- 172.21.15.6:0/3820350022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8aec101e40 0x7f8aec102cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:15.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.062+0000 7f8af1a7e700 1 -- 172.21.15.6:0/3820350022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8aec101e40 msgr2=0x7f8aec102cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:15.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.062+0000 7f8af1a7e700 1 --2- 172.21.15.6:0/3820350022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8aec101e40 0x7f8aec102cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:15.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.062+0000 7f8af1a7e700 1 -- 172.21.15.6:0/3820350022 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8adc005040 con 0x7f8aec1027e0 2024-02-09T21:57:15.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.062+0000 7f8af1a7e700 1 --2- 172.21.15.6:0/3820350022 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aec1027e0 0x7f8aec103200 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8ae8002a50 tx=0x7f8ae8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:57:15.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.063+0000 7f8ae2ffd700 1 -- 172.21.15.6:0/3820350022 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ae8007cc0 con 0x7f8aec1027e0 2024-02-09T21:57:15.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.063+0000 7f8ae2ffd700 1 -- 172.21.15.6:0/3820350022 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ae800f040 con 0x7f8aec1027e0 2024-02-09T21:57:15.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.063+0000 7f8af44e3700 1 -- 172.21.15.6:0/3820350022 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aec119ec0 con 0x7f8aec1027e0 2024-02-09T21:57:15.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.063+0000 7f8ae2ffd700 1 -- 172.21.15.6:0/3820350022 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ae800d9c0 con 0x7f8aec1027e0 2024-02-09T21:57:15.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.063+0000 7f8af44e3700 1 -- 172.21.15.6:0/3820350022 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aec11a370 con 0x7f8aec1027e0 2024-02-09T21:57:15.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.063+0000 7f8af44e3700 1 -- 172.21.15.6:0/3820350022 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aec100160 con 0x7f8aec1027e0 2024-02-09T21:57:15.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.064+0000 7f8ae2ffd700 1 -- 172.21.15.6:0/3820350022 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f8ae8009730 con 0x7f8aec1027e0 2024-02-09T21:57:15.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.064+0000 7f8ae2ffd700 1 --2- 172.21.15.6:0/3820350022 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f8ad80692a0 0x7f8ad806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:15.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.065+0000 7f8ae2ffd700 1 -- 172.21.15.6:0/3820350022 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f8ae8086e60 con 0x7f8aec1027e0 2024-02-09T21:57:15.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.065+0000 7f8af227f700 1 --2- 172.21.15.6:0/3820350022 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f8ad80692a0 0x7f8ad806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:15.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.065+0000 7f8af227f700 1 --2- 172.21.15.6:0/3820350022 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f8ad80692a0 0x7f8ad806b750 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8adc004f90 tx=0x7f8adc009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:57:15.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.069+0000 7f8ae2ffd700 1 -- 172.21.15.6:0/3820350022 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f8ae8053190 con 0x7f8aec1027e0 2024-02-09T21:57:15.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.306+0000 7f8af44e3700 1 -- 172.21.15.6:0/3820350022 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8aec103b30 con 0x7f8aec1027e0 2024-02-09T21:57:15.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.307+0000 7f8ae2ffd700 1 -- 172.21.15.6:0/3820350022 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) v1 ==== 53+0+83 (secure 0 0 0) 0x7f8aec103b30 con 0x7f8aec1027e0 2024-02-09T21:57:15.311 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-02-09T21:57:15.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.309+0000 7f8af44e3700 1 -- 172.21.15.6:0/3820350022 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f8ad80692a0 msgr2=0x7f8ad806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:15.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.309+0000 7f8af44e3700 1 --2- 172.21.15.6:0/3820350022 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f8ad80692a0 0x7f8ad806b750 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8adc004f90 tx=0x7f8adc009e30 comp rx=0 tx=0).stop 2024-02-09T21:57:15.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.309+0000 7f8af44e3700 1 -- 172.21.15.6:0/3820350022 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aec1027e0 msgr2=0x7f8aec103200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:15.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.309+0000 7f8af44e3700 1 --2- 172.21.15.6:0/3820350022 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aec1027e0 0x7f8aec103200 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8ae8002a50 tx=0x7f8ae8004600 comp rx=0 tx=0).stop 2024-02-09T21:57:15.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.309+0000 7f8af44e3700 1 -- 172.21.15.6:0/3820350022 shutdown_connections 2024-02-09T21:57:15.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.309+0000 7f8af44e3700 1 --2- 172.21.15.6:0/3820350022 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f8ad80692a0 0x7f8ad806b750 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:15.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.309+0000 7f8af44e3700 1 --2- 172.21.15.6:0/3820350022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8aec101e40 0x7f8aec102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:15.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.309+0000 7f8af44e3700 1 --2- 172.21.15.6:0/3820350022 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aec1027e0 0x7f8aec103200 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:15.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.309+0000 7f8af44e3700 1 -- 172.21.15.6:0/3820350022 >> 172.21.15.6:0/3820350022 conn(0x7f8aec071d90 msgr2=0x7f8aec105690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:57:15.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.309+0000 7f8af44e3700 1 -- 172.21.15.6:0/3820350022 shutdown_connections 2024-02-09T21:57:15.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:15.309+0000 7f8af44e3700 1 -- 172.21.15.6:0/3820350022 wait complete. 2024-02-09T21:57:15.918 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-02-09T21:57:15.918 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-09T21:57:15.918 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi090.front.sepia.ceph.com 2024-02-09T21:57:15.918 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-02-09T21:57:15.918 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-02-09T21:57:15.918 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-02-09T21:57:15.918 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-09T21:57:15.918 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-09T21:57:15.918 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-02-09T21:57:15.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:15.919 DEBUG:teuthology.orchestra.run.smithi090:> ip addr 2024-02-09T21:57:15.938 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-09T21:57:15.938 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-09T21:57:15.938 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-02-09T21:57:15.938 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-02-09T21:57:15.938 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-02-09T21:57:15.938 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-02-09T21:57:15.938 INFO:teuthology.orchestra.run.smithi090.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-09T21:57:15.938 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a8 brd ff:ff:ff:ff:ff:ff 2024-02-09T21:57:15.938 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f0 2024-02-09T21:57:15.938 INFO:teuthology.orchestra.run.smithi090.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-09T21:57:15.938 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a9 brd ff:ff:ff:ff:ff:ff 2024-02-09T21:57:15.939 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f1 2024-02-09T21:57:15.939 INFO:teuthology.orchestra.run.smithi090.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-09T21:57:15.939 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:66 brd ff:ff:ff:ff:ff:ff 2024-02-09T21:57:15.939 INFO:teuthology.orchestra.run.smithi090.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-09T21:57:15.939 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:67 brd ff:ff:ff:ff:ff:ff 2024-02-09T21:57:15.939 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.21.15.90/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-02-09T21:57:15.939 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft 41814sec preferred_lft 41814sec 2024-02-09T21:57:15.939 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::ec4:7aff:fe8f:d767/64 scope link 2024-02-09T21:57:15.939 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-02-09T21:57:15.939 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-02-09T21:57:15.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:15.940 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-02-09T21:57:15.940 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-02-09T21:57:15.940 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link add name ceph-brx type bridge 2024-02-09T21:57:15.940 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip addr flush dev ceph-brx 2024-02-09T21:57:15.940 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set ceph-brx up 2024-02-09T21:57:15.940 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-02-09T21:57:15.940 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-02-09T21:57:16.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:16.106 DEBUG:teuthology.orchestra.run.smithi090:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-09T21:57:16.134 INFO:teuthology.orchestra.run.smithi090.stdout:1 2024-02-09T21:57:16.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:16.135 DEBUG:teuthology.orchestra.run.smithi090:> route 2024-02-09T21:57:16.205 INFO:teuthology.orchestra.run.smithi090.stdout:Kernel IP routing table 2024-02-09T21:57:16.206 INFO:teuthology.orchestra.run.smithi090.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-09T21:57:16.206 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-02-09T21:57:16.206 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-02-09T21:57:16.206 INFO:teuthology.orchestra.run.smithi090.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-09T21:57:16.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:16.206 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-02-09T21:57:16.206 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-02-09T21:57:16.206 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-02-09T21:57:16.207 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-02-09T21:57:16.207 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-02-09T21:57:16.207 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-02-09T21:57:16.257 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:15 smithi090 ceph-mon[36996]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-02-09T21:57:16.257 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:15 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/3820350022' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-09T21:57:16.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:15 smithi006 ceph-mon[29258]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-02-09T21:57:16.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:15 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/3820350022' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-09T21:57:16.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:16.466 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-02-09T21:57:16.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:16.522 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list-id 2024-02-09T21:57:16.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:16.576 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-02-09T21:57:16.576 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-02-09T21:57:16.576 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-02-09T21:57:16.576 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-02-09T21:57:16.577 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-02-09T21:57:16.661 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-02-09T21:57:16.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:16.661 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-02-09T21:57:16.661 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-02-09T21:57:16.661 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-02-09T21:57:16.662 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-02-09T21:57:16.662 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-02-09T21:57:16.662 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-02-09T21:57:16.662 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-02-09T21:57:16.662 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-02-09T21:57:16.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:16.762 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-02-09T21:57:16.762 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-02-09T21:57:16.762 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set brx.0 up 2024-02-09T21:57:16.762 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set dev brx.0 master ceph-brx 2024-02-09T21:57:16.762 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-02-09T21:57:16.804 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-09T21:57:16.804 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-02-09T21:57:16.859 INFO:teuthology.orchestra.run.smithi090.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-02-09T21:57:16.859 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-09T21:57:16.859 DEBUG:teuthology.orchestra.run.smithi090:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-02-09T21:57:16.914 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-09T21:57:16.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:16.914 DEBUG:teuthology.orchestra.run.smithi090:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-09T21:57:17.009 INFO:teuthology.orchestra.run.smithi090.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-09T21:57:17.009 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-02-09T21:57:17.009 INFO:teuthology.orchestra.run.smithi090.stdout:invalid new device string format 2024-02-09T21:57:17.133 INFO:teuthology.orchestra.run.smithi090.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-09T21:57:17.134 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-02-09T21:57:17.134 INFO:teuthology.orchestra.run.smithi090.stdout:invalid new device string format 2024-02-09T21:57:17.134 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.90:3300" 2024-02-09T21:57:17.134 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-09T21:57:17.139 INFO:teuthology.orchestra.run.smithi090.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-09T21:57:17.139 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-02-09T21:57:17.139 INFO:teuthology.orchestra.run.smithi090.stdout:invalid new device string format 2024-02-09T21:57:17.139 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.90:3300" 2024-02-09T21:57:17.139 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-09T21:57:17.139 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: resolved to: "172.21.15.6:6789,172.21.15.90:6789" 2024-02-09T21:57:17.139 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.6:6789,172.21.15.90:6789:/ 2024-02-09T21:57:17.139 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=ee26d820-c794-11ee-95b8-87774f69a715" will pass to kernel 2024-02-09T21:57:17.143 INFO:teuthology.orchestra.run.smithi090.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-02-09T21:57:17.143 INFO:teuthology.orchestra.run.smithi090.stdout: You just mounted a file system that supports labels which does not 2024-02-09T21:57:17.143 INFO:teuthology.orchestra.run.smithi090.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-02-09T21:57:17.143 INFO:teuthology.orchestra.run.smithi090.stdout: applications will generate AVC messages and not be allowed access to 2024-02-09T21:57:17.143 INFO:teuthology.orchestra.run.smithi090.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-02-09T21:57:17.144 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-09T21:57:17.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:17.145 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-02-09T21:57:17.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:17.171 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-09T21:57:17.228 INFO:teuthology.orchestra.run.smithi090.stdout:client14526 2024-02-09T21:57:17.228 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-09T21:57:17.228 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-09T21:57:17.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:17.285 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-09T21:57:17.341 INFO:teuthology.orchestra.run.smithi090.stdout:client14526 2024-02-09T21:57:17.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:17.342 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-09T21:57:17.397 INFO:teuthology.orchestra.run.smithi090.stdout:client14526 2024-02-09T21:57:17.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T21:57:17.398 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/ee26d820-c794-11ee-95b8-87774f69a715.client14526/status') 2024-02-09T21:57:17.463 INFO:teuthology.orchestra.run.smithi090.stdout:instance: client.14526 (3)172.21.15.90:0/643119068 2024-02-09T21:57:17.464 INFO:teuthology.orchestra.run.smithi090.stdout:blocklisted: false 2024-02-09T21:57:17.464 INFO:teuthology.orchestra.run.smithi090.stderr:0+1 records in 2024-02-09T21:57:17.464 INFO:teuthology.orchestra.run.smithi090.stderr:0+1 records out 2024-02-09T21:57:17.464 INFO:teuthology.orchestra.run.smithi090.stderr:70 bytes copied, 3.6364e-05 s, 1.9 MB/s 2024-02-09T21:57:17.465 INFO:teuthology.run_tasks:Running task print... 2024-02-09T21:57:17.477 INFO:teuthology.task.print:**** done client 2024-02-09T21:57:17.478 INFO:teuthology.run_tasks:Running task parallel... 2024-02-09T21:57:17.528 INFO:teuthology.task.parallel:starting parallel... 2024-02-09T21:57:17.529 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-02-09T21:57:17.529 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-02-09T21:57:17.530 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi006.front.sepia.ceph.com 2024-02-09T21:57:17.530 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-02-09T21:57:17.530 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-02-09T21:57:17.530 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-02-09T21:57:17.534 INFO:tasks.workunit:Pulling workunits from ref 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T21:57:17.534 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-02-09T21:57:17.534 INFO:tasks.workunit:timeout=3h 2024-02-09T21:57:17.534 INFO:tasks.workunit:cleanup=True 2024-02-09T21:57:17.535 DEBUG:teuthology.orchestra.run.smithi006:> stat -- /home/ubuntu/cephtest/mnt.0 2024-02-09T21:57:17.554 INFO:teuthology.orchestra.run.smithi006.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-02-09T21:57:17.554 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-02-09T21:57:17.554 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 94h/148d Inode: 1 Links: 2 2024-02-09T21:57:17.554 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T21:57:17.554 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-02-09T21:57:17.554 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-02-09T21:57:17.554 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-02-09 21:56:42.889816061 +0000 2024-02-09T21:57:17.554 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-02-09 21:57:12.841946001 +0000 2024-02-09T21:57:17.555 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-02-09 21:56:42.889816061 +0000 2024-02-09T21:57:17.555 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-02-09T21:57:17.555 DEBUG:teuthology.orchestra.run.smithi006:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-02-09T21:57:17.627 DEBUG:teuthology.orchestra.run.smithi090:> stat -- /home/ubuntu/cephtest/mnt.1 2024-02-09T21:57:17.643 INFO:teuthology.orchestra.run.smithi090.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-02-09T21:57:17.643 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2024-02-09T21:57:17.643 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 73h/115d Inode: 1 Links: 3 2024-02-09T21:57:17.643 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T21:57:17.643 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-02-09T21:57:17.643 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-02-09T21:57:17.643 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-02-09 21:57:17.618894319 +0000 2024-02-09T21:57:17.643 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-02-09 21:57:17.618894319 +0000 2024-02-09T21:57:17.643 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-02-09 21:56:42.889816061 +0000 2024-02-09T21:57:17.643 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-02-09T21:57:17.643 DEBUG:teuthology.orchestra.run.smithi090:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-02-09T21:57:17.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:17 smithi006 ceph-mon[29258]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-09T21:57:17.704 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:17 smithi090 ceph-mon[36996]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-09T21:57:17.714 DEBUG:teuthology.orchestra.run.smithi006:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T21:57:17.715 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T21:57:17.730 INFO:tasks.workunit.client.0.smithi006.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-02-09T21:57:17.769 INFO:tasks.workunit.client.1.smithi090.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-02-09T21:57:17.815 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:57:19.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.444+0000 7f2be76d5700 1 -- 172.21.15.6:0/1391549949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2be01000e0 msgr2=0x7f2be0102520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:19.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.444+0000 7f2be76d5700 1 --2- 172.21.15.6:0/1391549949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2be01000e0 0x7f2be0102520 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4004940 tx=0x7f2bd4009e30 comp rx=0 tx=0).stop 2024-02-09T21:57:19.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.444+0000 7f2be76d5700 1 -- 172.21.15.6:0/1391549949 shutdown_connections 2024-02-09T21:57:19.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.444+0000 7f2be76d5700 1 --2- 172.21.15.6:0/1391549949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2be01000e0 0x7f2be0102520 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:19.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.444+0000 7f2be76d5700 1 --2- 172.21.15.6:0/1391549949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2be0066c50 0x7f2be00ffb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:19.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.444+0000 7f2be76d5700 1 -- 172.21.15.6:0/1391549949 >> 172.21.15.6:0/1391549949 conn(0x7f2be00f7520 msgr2=0x7f2be00f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:57:19.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.444+0000 7f2be76d5700 1 -- 172.21.15.6:0/1391549949 shutdown_connections 2024-02-09T21:57:19.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.444+0000 7f2be76d5700 1 -- 172.21.15.6:0/1391549949 wait complete. 2024-02-09T21:57:19.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.445+0000 7f2be76d5700 1 Processor -- start 2024-02-09T21:57:19.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.445+0000 7f2be76d5700 1 -- start start 2024-02-09T21:57:19.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.445+0000 7f2be76d5700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2be0066c50 0x7f2be00fd590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:19.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.445+0000 7f2be76d5700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2be01000e0 0x7f2be00fba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:19.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.445+0000 7f2be76d5700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be00fc010 con 0x7f2be0066c50 2024-02-09T21:57:19.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.445+0000 7f2be76d5700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be00fc180 con 0x7f2be01000e0 2024-02-09T21:57:19.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.445+0000 7f2be5471700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2be0066c50 0x7f2be00fd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:19.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.445+0000 7f2be5471700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2be0066c50 0x7f2be00fd590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35492/0 (socket says 172.21.15.6:35492) 2024-02-09T21:57:19.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.445+0000 7f2be5471700 1 -- 172.21.15.6:0/66561523 learned_addr learned my addr 172.21.15.6:0/66561523 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:57:19.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.446+0000 7f2be4c70700 1 --2- 172.21.15.6:0/66561523 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2be01000e0 0x7f2be00fba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:19.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.446+0000 7f2be5471700 1 -- 172.21.15.6:0/66561523 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2be01000e0 msgr2=0x7f2be00fba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:19.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.446+0000 7f2be5471700 1 --2- 172.21.15.6:0/66561523 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2be01000e0 0x7f2be00fba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:19.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.446+0000 7f2be5471700 1 -- 172.21.15.6:0/66561523 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd4005040 con 0x7f2be0066c50 2024-02-09T21:57:19.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.446+0000 7f2be5471700 1 --2- 172.21.15.6:0/66561523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2be0066c50 0x7f2be00fd590 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2bd000ec90 tx=0x7f2bd000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:57:19.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.446+0000 7f2bddffb700 1 -- 172.21.15.6:0/66561523 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bd000eea0 con 0x7f2be0066c50 2024-02-09T21:57:19.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.446+0000 7f2bddffb700 1 -- 172.21.15.6:0/66561523 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2bd000f040 con 0x7f2be0066c50 2024-02-09T21:57:19.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.446+0000 7f2be76d5700 1 -- 172.21.15.6:0/66561523 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2be00fc460 con 0x7f2be0066c50 2024-02-09T21:57:19.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.446+0000 7f2bddffb700 1 -- 172.21.15.6:0/66561523 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bd0009640 con 0x7f2be0066c50 2024-02-09T21:57:19.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.446+0000 7f2be76d5700 1 -- 172.21.15.6:0/66561523 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2be0115910 con 0x7f2be0066c50 2024-02-09T21:57:19.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.447+0000 7f2be76d5700 1 -- 172.21.15.6:0/66561523 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2be004bed0 con 0x7f2be0066c50 2024-02-09T21:57:19.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.448+0000 7f2bddffb700 1 -- 172.21.15.6:0/66561523 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f2bd000c940 con 0x7f2be0066c50 2024-02-09T21:57:19.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.448+0000 7f2bddffb700 1 --2- 172.21.15.6:0/66561523 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f2bc80692a0 0x7f2bc806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:19.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.448+0000 7f2bddffb700 1 -- 172.21.15.6:0/66561523 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f2bd0087640 con 0x7f2be0066c50 2024-02-09T21:57:19.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.448+0000 7f2be4c70700 1 --2- 172.21.15.6:0/66561523 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f2bc80692a0 0x7f2bc806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:19.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.448+0000 7f2be4c70700 1 --2- 172.21.15.6:0/66561523 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f2bc80692a0 0x7f2bc806b750 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2bd400b0d0 tx=0x7f2bd4009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:57:19.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.452+0000 7f2bddffb700 1 -- 172.21.15.6:0/66561523 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2bd0057400 con 0x7f2be0066c50 2024-02-09T21:57:19.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.646+0000 7f2be76d5700 1 -- 172.21.15.6:0/66561523 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f2be0105a10 con 0x7f2be0066c50 2024-02-09T21:57:19.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.647+0000 7f2bddffb700 1 -- 172.21.15.6:0/66561523 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7f2bd0056f90 con 0x7f2be0066c50 2024-02-09T21:57:19.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.649+0000 7f2be76d5700 1 -- 172.21.15.6:0/66561523 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f2bc80692a0 msgr2=0x7f2bc806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:19.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.649+0000 7f2be76d5700 1 --2- 172.21.15.6:0/66561523 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f2bc80692a0 0x7f2bc806b750 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2bd400b0d0 tx=0x7f2bd4009e30 comp rx=0 tx=0).stop 2024-02-09T21:57:19.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.649+0000 7f2be76d5700 1 -- 172.21.15.6:0/66561523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2be0066c50 msgr2=0x7f2be00fd590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:19.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.649+0000 7f2be76d5700 1 --2- 172.21.15.6:0/66561523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2be0066c50 0x7f2be00fd590 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2bd000ec90 tx=0x7f2bd000b040 comp rx=0 tx=0).stop 2024-02-09T21:57:19.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.649+0000 7f2be76d5700 1 -- 172.21.15.6:0/66561523 shutdown_connections 2024-02-09T21:57:19.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.650+0000 7f2be76d5700 1 --2- 172.21.15.6:0/66561523 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f2bc80692a0 0x7f2bc806b750 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:19.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.650+0000 7f2be76d5700 1 --2- 172.21.15.6:0/66561523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2be0066c50 0x7f2be00fd590 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:19.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.650+0000 7f2be76d5700 1 --2- 172.21.15.6:0/66561523 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2be01000e0 0x7f2be00fba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:19.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.650+0000 7f2be76d5700 1 -- 172.21.15.6:0/66561523 >> 172.21.15.6:0/66561523 conn(0x7f2be00f7520 msgr2=0x7f2be00f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:57:19.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.650+0000 7f2be76d5700 1 -- 172.21.15.6:0/66561523 shutdown_connections 2024-02-09T21:57:19.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:19.650+0000 7f2be76d5700 1 -- 172.21.15.6:0/66561523 wait complete. 2024-02-09T21:57:20.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:19 smithi090 ceph-mon[36996]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.6 KiB/s rd, 170 B/s wr, 4 op/s 2024-02-09T21:57:20.231 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:19 smithi006 ceph-mon[29258]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.6 KiB/s rd, 170 B/s wr, 4 op/s 2024-02-09T21:57:20.256 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-02-09T21:57:20.558 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:57:22.192 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:21 smithi006 ceph-mon[29258]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-02-09T21:57:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.188+0000 7f2849281700 1 -- 172.21.15.6:0/3396755242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28440fcef0 msgr2=0x7f28440fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:22.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.188+0000 7f2849281700 1 --2- 172.21.15.6:0/3396755242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28440fcef0 0x7f28440fd320 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2834004940 tx=0x7f2834009e30 comp rx=0 tx=0).stop 2024-02-09T21:57:22.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.188+0000 7f2849281700 1 -- 172.21.15.6:0/3396755242 shutdown_connections 2024-02-09T21:57:22.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.188+0000 7f2849281700 1 --2- 172.21.15.6:0/3396755242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28440fe0e0 0x7f28440fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:22.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.188+0000 7f2849281700 1 --2- 172.21.15.6:0/3396755242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28440fcef0 0x7f28440fd320 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:22.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.188+0000 7f2849281700 1 -- 172.21.15.6:0/3396755242 >> 172.21.15.6:0/3396755242 conn(0x7f28440f8620 msgr2=0x7f28440faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:57:22.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.188+0000 7f2849281700 1 -- 172.21.15.6:0/3396755242 shutdown_connections 2024-02-09T21:57:22.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.188+0000 7f2849281700 1 -- 172.21.15.6:0/3396755242 wait complete. 2024-02-09T21:57:22.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.189+0000 7f2849281700 1 Processor -- start 2024-02-09T21:57:22.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.189+0000 7f2849281700 1 -- start start 2024-02-09T21:57:22.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.189+0000 7f2849281700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28440fcef0 0x7f284410d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:22.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.189+0000 7f2849281700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28440fe0e0 0x7f284410de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:22.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.189+0000 7f2849281700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284410e430 con 0x7f28440fcef0 2024-02-09T21:57:22.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.189+0000 7f2849281700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284410ecd0 con 0x7f28440fe0e0 2024-02-09T21:57:22.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.189+0000 7f2842ffd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28440fcef0 0x7f284410d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:22.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.190+0000 7f2842ffd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28440fcef0 0x7f284410d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35510/0 (socket says 172.21.15.6:35510) 2024-02-09T21:57:22.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.190+0000 7f28427fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28440fe0e0 0x7f284410de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:22.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.190+0000 7f2842ffd700 1 -- 172.21.15.6:0/3254395781 learned_addr learned my addr 172.21.15.6:0/3254395781 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:57:22.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.190+0000 7f2842ffd700 1 -- 172.21.15.6:0/3254395781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28440fe0e0 msgr2=0x7f284410de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:22.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.190+0000 7f2842ffd700 1 --2- 172.21.15.6:0/3254395781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28440fe0e0 0x7f284410de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:22.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.190+0000 7f2842ffd700 1 -- 172.21.15.6:0/3254395781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2834005040 con 0x7f28440fcef0 2024-02-09T21:57:22.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.190+0000 7f28427fc700 1 --2- 172.21.15.6:0/3254395781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28440fe0e0 0x7f284410de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T21:57:22.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.190+0000 7f2842ffd700 1 --2- 172.21.15.6:0/3254395781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28440fcef0 0x7f284410d8f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2834000c00 tx=0x7f2834009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:57:22.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.190+0000 7f282b7fe700 1 -- 172.21.15.6:0/3254395781 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f283400e070 con 0x7f28440fcef0 2024-02-09T21:57:22.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.190+0000 7f282b7fe700 1 -- 172.21.15.6:0/3254395781 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f283401c400 con 0x7f28440fcef0 2024-02-09T21:57:22.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.190+0000 7f282b7fe700 1 -- 172.21.15.6:0/3254395781 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f283400b670 con 0x7f28440fcef0 2024-02-09T21:57:22.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.190+0000 7f2849281700 1 -- 172.21.15.6:0/3254395781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f284410ef50 con 0x7f28440fcef0 2024-02-09T21:57:22.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.190+0000 7f2849281700 1 -- 172.21.15.6:0/3254395781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f284410f430 con 0x7f28440fcef0 2024-02-09T21:57:22.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.191+0000 7f2849281700 1 -- 172.21.15.6:0/3254395781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f284404bed0 con 0x7f28440fcef0 2024-02-09T21:57:22.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.192+0000 7f282b7fe700 1 -- 172.21.15.6:0/3254395781 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f283401c9e0 con 0x7f28440fcef0 2024-02-09T21:57:22.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.193+0000 7f282b7fe700 1 --2- 172.21.15.6:0/3254395781 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f282c0692a0 0x7f282c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:22.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.193+0000 7f282b7fe700 1 -- 172.21.15.6:0/3254395781 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f28340886d0 con 0x7f28440fcef0 2024-02-09T21:57:22.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.193+0000 7f28427fc700 1 --2- 172.21.15.6:0/3254395781 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f282c0692a0 0x7f282c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:22.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.193+0000 7f28427fc700 1 --2- 172.21.15.6:0/3254395781 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f282c0692a0 0x7f282c06b750 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2838009b80 tx=0x7f283800b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:57:22.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:21 smithi090 ceph-mon[36996]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-02-09T21:57:22.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.202+0000 7f282b7fe700 1 -- 172.21.15.6:0/3254395781 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2834058530 con 0x7f28440fcef0 2024-02-09T21:57:22.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.404+0000 7f2849281700 1 -- 172.21.15.6:0/3254395781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f2844105a30 con 0x7f28440fcef0 2024-02-09T21:57:22.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.404+0000 7f282b7fe700 1 -- 172.21.15.6:0/3254395781 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f28340580c0 con 0x7f28440fcef0 2024-02-09T21:57:22.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.406+0000 7f2849281700 1 -- 172.21.15.6:0/3254395781 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f282c0692a0 msgr2=0x7f282c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:22.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.406+0000 7f2849281700 1 --2- 172.21.15.6:0/3254395781 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f282c0692a0 0x7f282c06b750 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2838009b80 tx=0x7f283800b040 comp rx=0 tx=0).stop 2024-02-09T21:57:22.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.406+0000 7f2849281700 1 -- 172.21.15.6:0/3254395781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28440fcef0 msgr2=0x7f284410d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:22.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.406+0000 7f2849281700 1 --2- 172.21.15.6:0/3254395781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28440fcef0 0x7f284410d8f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2834000c00 tx=0x7f2834009e30 comp rx=0 tx=0).stop 2024-02-09T21:57:22.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.406+0000 7f2849281700 1 -- 172.21.15.6:0/3254395781 shutdown_connections 2024-02-09T21:57:22.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.406+0000 7f2849281700 1 --2- 172.21.15.6:0/3254395781 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f282c0692a0 0x7f282c06b750 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:22.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.406+0000 7f2849281700 1 --2- 172.21.15.6:0/3254395781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28440fcef0 0x7f284410d8f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:22.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.406+0000 7f2849281700 1 --2- 172.21.15.6:0/3254395781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28440fe0e0 0x7f284410de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:22.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.406+0000 7f2849281700 1 -- 172.21.15.6:0/3254395781 >> 172.21.15.6:0/3254395781 conn(0x7f28440f8620 msgr2=0x7f28441013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:57:22.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.406+0000 7f2849281700 1 -- 172.21.15.6:0/3254395781 shutdown_connections 2024-02-09T21:57:22.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:22.406+0000 7f2849281700 1 -- 172.21.15.6:0/3254395781 wait complete. 2024-02-09T21:57:23.155 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph config set global log_to_journald false --force' 2024-02-09T21:57:23.439 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:57:24.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:23 smithi006 ceph-mon[29258]: pgmap v196: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 682 B/s wr, 5 op/s 2024-02-09T21:57:24.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:23 smithi090 ceph-mon[36996]: pgmap v196: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 682 B/s wr, 5 op/s 2024-02-09T21:57:25.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.142+0000 7febaa19d700 1 -- 172.21.15.6:0/434089728 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feba40fbc40 msgr2=0x7feba40fc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:25.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.142+0000 7febaa19d700 1 --2- 172.21.15.6:0/434089728 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feba40fbc40 0x7feba40fc050 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7feba0004940 tx=0x7feba0009e30 comp rx=0 tx=0).stop 2024-02-09T21:57:25.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.142+0000 7febaa19d700 1 -- 172.21.15.6:0/434089728 shutdown_connections 2024-02-09T21:57:25.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.142+0000 7febaa19d700 1 --2- 172.21.15.6:0/434089728 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feba40fce50 0x7feba40fd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:25.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.142+0000 7febaa19d700 1 --2- 172.21.15.6:0/434089728 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feba40fbc40 0x7feba40fc050 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:25.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.142+0000 7febaa19d700 1 -- 172.21.15.6:0/434089728 >> 172.21.15.6:0/434089728 conn(0x7feba40f72c0 msgr2=0x7feba40f9730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:57:25.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.142+0000 7febaa19d700 1 -- 172.21.15.6:0/434089728 shutdown_connections 2024-02-09T21:57:25.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.142+0000 7febaa19d700 1 -- 172.21.15.6:0/434089728 wait complete. 2024-02-09T21:57:25.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.143+0000 7febaa19d700 1 Processor -- start 2024-02-09T21:57:25.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.143+0000 7febaa19d700 1 -- start start 2024-02-09T21:57:25.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.143+0000 7febaa19d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feba40fbc40 0x7feba410f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:25.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.143+0000 7febaa19d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feba40fce50 0x7feba410fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:25.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.143+0000 7febaa19d700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba41104f0 con 0x7feba40fbc40 2024-02-09T21:57:25.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.143+0000 7febaa19d700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba4110d90 con 0x7feba40fce50 2024-02-09T21:57:25.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.143+0000 7feba919b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feba40fbc40 0x7feba410f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:25.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.143+0000 7feba899a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feba40fce50 0x7feba410fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:25.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.143+0000 7feba919b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feba40fbc40 0x7feba410f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35534/0 (socket says 172.21.15.6:35534) 2024-02-09T21:57:25.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.143+0000 7feba899a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feba40fce50 0x7feba410fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.6:35956/0 (socket says 172.21.15.6:35956) 2024-02-09T21:57:25.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.143+0000 7feba919b700 1 -- 172.21.15.6:0/1117074557 learned_addr learned my addr 172.21.15.6:0/1117074557 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:57:25.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.144+0000 7feba899a700 1 -- 172.21.15.6:0/1117074557 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feba40fbc40 msgr2=0x7feba410f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:25.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.144+0000 7feba899a700 1 --2- 172.21.15.6:0/1117074557 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feba40fbc40 0x7feba410f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:25.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.144+0000 7feba899a700 1 -- 172.21.15.6:0/1117074557 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba0005040 con 0x7feba40fce50 2024-02-09T21:57:25.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.144+0000 7feba899a700 1 --2- 172.21.15.6:0/1117074557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feba40fce50 0x7feba410fef0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7feb98009fd0 tx=0x7feb9800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:57:25.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.144+0000 7feb95ffb700 1 -- 172.21.15.6:0/1117074557 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb98004e90 con 0x7feba40fce50 2024-02-09T21:57:25.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.144+0000 7feb95ffb700 1 -- 172.21.15.6:0/1117074557 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb9800f040 con 0x7feba40fce50 2024-02-09T21:57:25.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.144+0000 7feb95ffb700 1 -- 172.21.15.6:0/1117074557 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb9800ca50 con 0x7feba40fce50 2024-02-09T21:57:25.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.144+0000 7febaa19d700 1 -- 172.21.15.6:0/1117074557 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba4111010 con 0x7feba40fce50 2024-02-09T21:57:25.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.144+0000 7febaa19d700 1 -- 172.21.15.6:0/1117074557 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feba4111500 con 0x7feba40fce50 2024-02-09T21:57:25.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.146+0000 7febaa19d700 1 -- 172.21.15.6:0/1117074557 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba404bed0 con 0x7feba40fce50 2024-02-09T21:57:25.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.146+0000 7feb95ffb700 1 -- 172.21.15.6:0/1117074557 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7feb98007cc0 con 0x7feba40fce50 2024-02-09T21:57:25.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.146+0000 7feb95ffb700 1 --2- 172.21.15.6:0/1117074557 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7feb900692a0 0x7feb9006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:25.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.147+0000 7feb95ffb700 1 -- 172.21.15.6:0/1117074557 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7feb98086c10 con 0x7feba40fce50 2024-02-09T21:57:25.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.147+0000 7feba919b700 1 --2- 172.21.15.6:0/1117074557 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7feb900692a0 0x7feb9006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:25.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.147+0000 7feba919b700 1 --2- 172.21.15.6:0/1117074557 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7feb900692a0 0x7feb9006b750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7feba000b0d0 tx=0x7feba0009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:57:25.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.151+0000 7feb95ffb700 1 -- 172.21.15.6:0/1117074557 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7feb9804dbd0 con 0x7feba40fce50 2024-02-09T21:57:25.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.376+0000 7febaa19d700 1 -- 172.21.15.6:0/1117074557 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7feba4111880 con 0x7feba40fce50 2024-02-09T21:57:25.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.377+0000 7feb95ffb700 1 -- 172.21.15.6:0/1117074557 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7feb980569a0 con 0x7feba40fce50 2024-02-09T21:57:25.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.379+0000 7febaa19d700 1 -- 172.21.15.6:0/1117074557 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7feb900692a0 msgr2=0x7feb9006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:25.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.379+0000 7febaa19d700 1 --2- 172.21.15.6:0/1117074557 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7feb900692a0 0x7feb9006b750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7feba000b0d0 tx=0x7feba0009e30 comp rx=0 tx=0).stop 2024-02-09T21:57:25.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.379+0000 7febaa19d700 1 -- 172.21.15.6:0/1117074557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feba40fce50 msgr2=0x7feba410fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:25.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.379+0000 7febaa19d700 1 --2- 172.21.15.6:0/1117074557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feba40fce50 0x7feba410fef0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7feb98009fd0 tx=0x7feb9800b040 comp rx=0 tx=0).stop 2024-02-09T21:57:25.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.379+0000 7febaa19d700 1 -- 172.21.15.6:0/1117074557 shutdown_connections 2024-02-09T21:57:25.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.379+0000 7febaa19d700 1 --2- 172.21.15.6:0/1117074557 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7feb900692a0 0x7feb9006b750 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:25.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.379+0000 7febaa19d700 1 --2- 172.21.15.6:0/1117074557 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feba40fbc40 0x7feba410f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:25.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.379+0000 7febaa19d700 1 --2- 172.21.15.6:0/1117074557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feba40fce50 0x7feba410fef0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:25.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.379+0000 7febaa19d700 1 -- 172.21.15.6:0/1117074557 >> 172.21.15.6:0/1117074557 conn(0x7feba40f72c0 msgr2=0x7feba40656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:57:25.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.380+0000 7febaa19d700 1 -- 172.21.15.6:0/1117074557 shutdown_connections 2024-02-09T21:57:25.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:25.380+0000 7febaa19d700 1 -- 172.21.15.6:0/1117074557 wait complete. 2024-02-09T21:57:26.072 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:25 smithi006 ceph-mon[29258]: pgmap v197: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 682 B/s wr, 4 op/s 2024-02-09T21:57:26.096 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-02-09T21:57:26.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:25 smithi090 ceph-mon[36996]: pgmap v197: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 682 B/s wr, 4 op/s 2024-02-09T21:57:26.374 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:57:27.622 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:27 smithi006 ceph-mon[29258]: pgmap v198: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-02-09T21:57:27.622 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:27 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:57:27.704 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:27 smithi090 ceph-mon[36996]: pgmap v198: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-02-09T21:57:27.704 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:27 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:57:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.165+0000 7f5dae798700 1 -- 172.21.15.6:0/477291222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5da80fbe40 msgr2=0x7f5da80fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.165+0000 7f5dae798700 1 --2- 172.21.15.6:0/477291222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5da80fbe40 0x7f5da80fc250 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5d98004940 tx=0x7f5d98009e30 comp rx=0 tx=0).stop 2024-02-09T21:57:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.165+0000 7f5dae798700 1 -- 172.21.15.6:0/477291222 shutdown_connections 2024-02-09T21:57:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.165+0000 7f5dae798700 1 --2- 172.21.15.6:0/477291222 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5da80fcfb0 0x7f5da80fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.165+0000 7f5dae798700 1 --2- 172.21.15.6:0/477291222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5da80fbe40 0x7f5da80fc250 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.165+0000 7f5dae798700 1 -- 172.21.15.6:0/477291222 >> 172.21.15.6:0/477291222 conn(0x7f5da80f7500 msgr2=0x7f5da80f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:57:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.166+0000 7f5dae798700 1 -- 172.21.15.6:0/477291222 shutdown_connections 2024-02-09T21:57:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.166+0000 7f5dae798700 1 -- 172.21.15.6:0/477291222 wait complete. 2024-02-09T21:57:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.166+0000 7f5dae798700 1 Processor -- start 2024-02-09T21:57:28.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.166+0000 7f5dae798700 1 -- start start 2024-02-09T21:57:28.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.167+0000 7f5dae798700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5da80fbe40 0x7f5da81105d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:28.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.167+0000 7f5dae798700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5da80fcfb0 0x7f5da8112b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:28.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.167+0000 7f5dae798700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da8066d10 con 0x7f5da80fcfb0 2024-02-09T21:57:28.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.167+0000 7f5dae798700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da8066e50 con 0x7f5da80fbe40 2024-02-09T21:57:28.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.167+0000 7f5da77fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5da80fcfb0 0x7f5da8112b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:28.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.167+0000 7f5da77fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5da80fcfb0 0x7f5da8112b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47490/0 (socket says 172.21.15.6:47490) 2024-02-09T21:57:28.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.167+0000 7f5da77fe700 1 -- 172.21.15.6:0/51792825 learned_addr learned my addr 172.21.15.6:0/51792825 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:57:28.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.167+0000 7f5da7fff700 1 --2- 172.21.15.6:0/51792825 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5da80fbe40 0x7f5da81105d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:28.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.167+0000 7f5da77fe700 1 -- 172.21.15.6:0/51792825 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5da80fbe40 msgr2=0x7f5da81105d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:57:28.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.167+0000 7f5da77fe700 1 --2- 172.21.15.6:0/51792825 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5da80fbe40 0x7f5da81105d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:57:28.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.167+0000 7f5da77fe700 1 -- 172.21.15.6:0/51792825 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d98005040 con 0x7f5da80fcfb0 2024-02-09T21:57:28.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.167+0000 7f5da77fe700 1 --2- 172.21.15.6:0/51792825 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5da80fcfb0 0x7f5da8112b20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c00eb40 tx=0x7f5d9c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:57:28.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.167+0000 7f5da4ff9700 1 -- 172.21.15.6:0/51792825 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d9c007cc0 con 0x7f5da80fcfb0 2024-02-09T21:57:28.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.168+0000 7f5dae798700 1 -- 172.21.15.6:0/51792825 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da8113120 con 0x7f5da80fcfb0 2024-02-09T21:57:28.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.168+0000 7f5da4ff9700 1 -- 172.21.15.6:0/51792825 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d9c00f040 con 0x7f5da80fcfb0 2024-02-09T21:57:28.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.168+0000 7f5dae798700 1 -- 172.21.15.6:0/51792825 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da8113610 con 0x7f5da80fcfb0 2024-02-09T21:57:28.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.168+0000 7f5da4ff9700 1 -- 172.21.15.6:0/51792825 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d9c00ca10 con 0x7f5da80fcfb0 2024-02-09T21:57:28.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.169+0000 7f5dae798700 1 -- 172.21.15.6:0/51792825 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d940052e0 con 0x7f5da80fcfb0 2024-02-09T21:57:28.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.169+0000 7f5da4ff9700 1 -- 172.21.15.6:0/51792825 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f5d9c00cc80 con 0x7f5da80fcfb0 2024-02-09T21:57:28.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.170+0000 7f5da4ff9700 1 --2- 172.21.15.6:0/51792825 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f5d900692a0 0x7f5d9006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:57:28.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.170+0000 7f5da4ff9700 1 -- 172.21.15.6:0/51792825 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f5d9c086d20 con 0x7f5da80fcfb0 2024-02-09T21:57:28.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.170+0000 7f5da7fff700 1 --2- 172.21.15.6:0/51792825 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f5d900692a0 0x7f5d9006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:57:28.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.170+0000 7f5da7fff700 1 --2- 172.21.15.6:0/51792825 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f5d900692a0 0x7f5d9006b750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5d98000c00 tx=0x7f5d98009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:57:28.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.174+0000 7f5da4ff9700 1 -- 172.21.15.6:0/51792825 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f5d9c04dbd0 con 0x7f5da80fcfb0 2024-02-09T21:57:28.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:57:28.461+0000 7f5dae798700 1 -- 172.21.15.6:0/51792825 --> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d94000ca0 con 0x7f5d900692a0 2024-02-09T21:57:30.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:29 smithi090 ceph-mon[36996]: from='client.14542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:57:30.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:29 smithi090 ceph-mon[36996]: pgmap v199: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.1 KiB/s rd, 767 B/s wr, 5 op/s 2024-02-09T21:57:30.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:29 smithi006 ceph-mon[29258]: from='client.14542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:57:30.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:29 smithi006 ceph-mon[29258]: pgmap v199: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.1 KiB/s rd, 767 B/s wr, 5 op/s 2024-02-09T21:57:31.992 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:31 smithi090 ceph-mon[36996]: pgmap v200: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 597 B/s wr, 3 op/s 2024-02-09T21:57:32.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:31 smithi006 ceph-mon[29258]: pgmap v200: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 597 B/s wr, 3 op/s 2024-02-09T21:57:34.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:33 smithi090 ceph-mon[36996]: pgmap v201: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.6 KiB/s rd, 597 B/s wr, 5 op/s 2024-02-09T21:57:34.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:33 smithi006 ceph-mon[29258]: pgmap v201: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.6 KiB/s rd, 597 B/s wr, 5 op/s 2024-02-09T21:57:36.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:35 smithi090 ceph-mon[36996]: pgmap v202: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 3 op/s 2024-02-09T21:57:36.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:35 smithi006 ceph-mon[29258]: pgmap v202: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 3 op/s 2024-02-09T21:57:37.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:37 smithi006 ceph-mon[29258]: pgmap v203: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 85 B/s wr, 3 op/s 2024-02-09T21:57:37.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:37 smithi090 ceph-mon[36996]: pgmap v203: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 85 B/s wr, 3 op/s 2024-02-09T21:57:40.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:39 smithi090 ceph-mon[36996]: pgmap v204: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-09T21:57:40.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:39 smithi006 ceph-mon[29258]: pgmap v204: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-09T21:57:42.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:41 smithi090 ceph-mon[36996]: pgmap v205: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-09T21:57:42.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:41 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:57:42.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:41 smithi006 ceph-mon[29258]: pgmap v205: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-09T21:57:42.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:41 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:57:44.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:43 smithi090 ceph-mon[36996]: pgmap v206: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-02-09T21:57:44.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:43 smithi006 ceph-mon[29258]: pgmap v206: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-02-09T21:57:46.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:45 smithi090 ceph-mon[36996]: pgmap v207: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-02-09T21:57:46.409 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:45 smithi006 ceph-mon[29258]: pgmap v207: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-02-09T21:57:47.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:47 smithi006 ceph-mon[29258]: pgmap v208: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-09T21:57:47.704 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:47 smithi090 ceph-mon[36996]: pgmap v208: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-09T21:57:50.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:49 smithi090 ceph-mon[36996]: pgmap v209: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-09T21:57:50.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:49 smithi006 ceph-mon[29258]: pgmap v209: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-09T21:57:52.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:51 smithi090 ceph-mon[36996]: pgmap v210: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-09T21:57:52.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:51 smithi006 ceph-mon[29258]: pgmap v210: 65 pgs: 65 active+clean; 585 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-09T21:57:54.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:53 smithi090 ceph-mon[36996]: pgmap v211: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 426 B/s wr, 4 op/s 2024-02-09T21:57:54.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:53 smithi006 ceph-mon[29258]: pgmap v211: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 426 B/s wr, 4 op/s 2024-02-09T21:57:56.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:55 smithi006 ceph-mon[29258]: pgmap v212: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 426 B/s wr, 4 op/s 2024-02-09T21:57:56.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:55 smithi090 ceph-mon[36996]: pgmap v212: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 426 B/s wr, 4 op/s 2024-02-09T21:57:57.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:57 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:57:57.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:57 smithi006 ceph-mon[29258]: pgmap v213: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 4 op/s 2024-02-09T21:57:57.704 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:57 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:57:57.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:57 smithi090 ceph-mon[36996]: pgmap v213: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 4 op/s 2024-02-09T21:58:00.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:57:59 smithi090 ceph-mon[36996]: pgmap v214: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-02-09T21:58:00.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:57:59 smithi006 ceph-mon[29258]: pgmap v214: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-02-09T21:58:00.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:00.479+0000 7f5da4ff9700 1 -- 172.21.15.6:0/51792825 <== mgr.14216 v2:172.21.15.6:6800/2624137529 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f5d94000ca0 con 0x7f5d900692a0 2024-02-09T21:58:00.483 INFO:teuthology.orchestra.run.smithi006.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T21:58:00.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:00.481+0000 7f5dae798700 1 -- 172.21.15.6:0/51792825 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f5d900692a0 msgr2=0x7f5d9006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:58:00.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:00.481+0000 7f5dae798700 1 --2- 172.21.15.6:0/51792825 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f5d900692a0 0x7f5d9006b750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5d98000c00 tx=0x7f5d98009e30 comp rx=0 tx=0).stop 2024-02-09T21:58:00.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:00.481+0000 7f5dae798700 1 -- 172.21.15.6:0/51792825 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5da80fcfb0 msgr2=0x7f5da8112b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:58:00.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:00.481+0000 7f5dae798700 1 --2- 172.21.15.6:0/51792825 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5da80fcfb0 0x7f5da8112b20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c00eb40 tx=0x7f5d9c00b040 comp rx=0 tx=0).stop 2024-02-09T21:58:00.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:00.481+0000 7f5dae798700 1 -- 172.21.15.6:0/51792825 shutdown_connections 2024-02-09T21:58:00.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:00.481+0000 7f5dae798700 1 --2- 172.21.15.6:0/51792825 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f5d900692a0 0x7f5d9006b750 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:00.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:00.481+0000 7f5dae798700 1 --2- 172.21.15.6:0/51792825 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5da80fbe40 0x7f5da81105d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:00.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:00.481+0000 7f5dae798700 1 --2- 172.21.15.6:0/51792825 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5da80fcfb0 0x7f5da8112b20 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:00.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:00.481+0000 7f5dae798700 1 -- 172.21.15.6:0/51792825 >> 172.21.15.6:0/51792825 conn(0x7f5da80f7500 msgr2=0x7f5da8100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:58:00.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:00.481+0000 7f5dae798700 1 -- 172.21.15.6:0/51792825 shutdown_connections 2024-02-09T21:58:00.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:00.481+0000 7f5dae798700 1 -- 172.21.15.6:0/51792825 wait complete. 2024-02-09T21:58:01.290 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-02-09T21:58:01.643 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T21:58:01.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:01 smithi006 ceph-mon[29258]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T21:58:01.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:58:01.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:58:01.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:01 smithi006 ceph-mon[29258]: pgmap v215: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-02-09T21:58:01.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:58:01.658 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:58:01.659 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:01 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:58:01.659 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:01 smithi006 ceph-mon[29258]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T21:58:01.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:01 smithi090 ceph-mon[36996]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T21:58:01.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:58:01.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:58:01.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:01 smithi090 ceph-mon[36996]: pgmap v215: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-02-09T21:58:01.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T21:58:01.705 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T21:58:01.706 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:01 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:58:01.706 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:01 smithi090 ceph-mon[36996]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T21:58:03.362 INFO:tasks.workunit.client.1.smithi090.stderr:Note: switching to '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'. 2024-02-09T21:58:03.362 INFO:tasks.workunit.client.1.smithi090.stderr: 2024-02-09T21:58:03.362 INFO:tasks.workunit.client.1.smithi090.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-09T21:58:03.362 INFO:tasks.workunit.client.1.smithi090.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-09T21:58:03.363 INFO:tasks.workunit.client.1.smithi090.stderr:state without impacting any branches by switching back to a branch. 2024-02-09T21:58:03.363 INFO:tasks.workunit.client.1.smithi090.stderr: 2024-02-09T21:58:03.363 INFO:tasks.workunit.client.1.smithi090.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-09T21:58:03.363 INFO:tasks.workunit.client.1.smithi090.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-09T21:58:03.363 INFO:tasks.workunit.client.1.smithi090.stderr: 2024-02-09T21:58:03.363 INFO:tasks.workunit.client.1.smithi090.stderr: git switch -c 2024-02-09T21:58:03.363 INFO:tasks.workunit.client.1.smithi090.stderr: 2024-02-09T21:58:03.363 INFO:tasks.workunit.client.1.smithi090.stderr:Or undo this operation with: 2024-02-09T21:58:03.363 INFO:tasks.workunit.client.1.smithi090.stderr: 2024-02-09T21:58:03.363 INFO:tasks.workunit.client.1.smithi090.stderr: git switch - 2024-02-09T21:58:03.363 INFO:tasks.workunit.client.1.smithi090.stderr: 2024-02-09T21:58:03.364 INFO:tasks.workunit.client.1.smithi090.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-09T21:58:03.364 INFO:tasks.workunit.client.1.smithi090.stderr: 2024-02-09T21:58:03.364 INFO:tasks.workunit.client.1.smithi090.stderr:HEAD is now at 290212a1bfd Merge pull request #53521 from xxhdx1985126/wip-62857 2024-02-09T21:58:03.366 DEBUG:teuthology.orchestra.run.smithi090:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-02-09T21:58:03.518 INFO:tasks.workunit.client.1.smithi090.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-02-09T21:58:03.520 INFO:tasks.workunit.client.1.smithi090.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-02-09T21:58:03.521 INFO:tasks.workunit.client.1.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-02-09T21:58:04.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.114+0000 7fe1b1006700 1 -- 172.21.15.6:0/892380978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1ac0fbe50 msgr2=0x7fe1ac0fc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:58:04.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.114+0000 7fe1b1006700 1 --2- 172.21.15.6:0/892380978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1ac0fbe50 0x7fe1ac0fc260 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe19c004940 tx=0x7fe19c009e30 comp rx=0 tx=0).stop 2024-02-09T21:58:04.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.114+0000 7fe1b1006700 1 -- 172.21.15.6:0/892380978 shutdown_connections 2024-02-09T21:58:04.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.114+0000 7fe1b1006700 1 --2- 172.21.15.6:0/892380978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1ac0fcfc0 0x7fe1ac0fd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:04.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.114+0000 7fe1b1006700 1 --2- 172.21.15.6:0/892380978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1ac0fbe50 0x7fe1ac0fc260 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:04.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.114+0000 7fe1b1006700 1 -- 172.21.15.6:0/892380978 >> 172.21.15.6:0/892380978 conn(0x7fe1ac0f7510 msgr2=0x7fe1ac0f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:58:04.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.114+0000 7fe1b1006700 1 -- 172.21.15.6:0/892380978 shutdown_connections 2024-02-09T21:58:04.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.114+0000 7fe1b1006700 1 -- 172.21.15.6:0/892380978 wait complete. 2024-02-09T21:58:04.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.115+0000 7fe1b1006700 1 Processor -- start 2024-02-09T21:58:04.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.115+0000 7fe1b1006700 1 -- start start 2024-02-09T21:58:04.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.116+0000 7fe1b1006700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1ac0fbe50 0x7fe1ac10c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:58:04.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.116+0000 7fe1b1006700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1ac0fcfc0 0x7fe1ac10cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:58:04.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.116+0000 7fe1b1006700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ac10d310 con 0x7fe1ac0fbe50 2024-02-09T21:58:04.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.116+0000 7fe1b1006700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ac10dbb0 con 0x7fe1ac0fcfc0 2024-02-09T21:58:04.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.116+0000 7fe1aad9d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1ac0fbe50 0x7fe1ac10c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:58:04.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.116+0000 7fe1aad9d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1ac0fbe50 0x7fe1ac10c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39450/0 (socket says 172.21.15.6:39450) 2024-02-09T21:58:04.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.116+0000 7fe1aad9d700 1 -- 172.21.15.6:0/2797516826 learned_addr learned my addr 172.21.15.6:0/2797516826 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:58:04.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.116+0000 7fe1aa59c700 1 --2- 172.21.15.6:0/2797516826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1ac0fcfc0 0x7fe1ac10cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:58:04.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.116+0000 7fe1aad9d700 1 -- 172.21.15.6:0/2797516826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1ac0fcfc0 msgr2=0x7fe1ac10cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:58:04.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.116+0000 7fe1aad9d700 1 --2- 172.21.15.6:0/2797516826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1ac0fcfc0 0x7fe1ac10cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:04.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.116+0000 7fe1aad9d700 1 -- 172.21.15.6:0/2797516826 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe19c005040 con 0x7fe1ac0fbe50 2024-02-09T21:58:04.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.116+0000 7fe1aa59c700 1 --2- 172.21.15.6:0/2797516826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1ac0fcfc0 0x7fe1ac10cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T21:58:04.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.117+0000 7fe1aad9d700 1 --2- 172.21.15.6:0/2797516826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1ac0fbe50 0x7fe1ac10c7d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe19c009fd0 tx=0x7fe19c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:58:04.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.117+0000 7fe1937fe700 1 -- 172.21.15.6:0/2797516826 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe19c00e070 con 0x7fe1ac0fbe50 2024-02-09T21:58:04.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.117+0000 7fe1937fe700 1 -- 172.21.15.6:0/2797516826 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe19c013d00 con 0x7fe1ac0fbe50 2024-02-09T21:58:04.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.117+0000 7fe1b1006700 1 -- 172.21.15.6:0/2797516826 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1ac10de30 con 0x7fe1ac0fbe50 2024-02-09T21:58:04.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.117+0000 7fe1937fe700 1 -- 172.21.15.6:0/2797516826 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe19c0236c0 con 0x7fe1ac0fbe50 2024-02-09T21:58:04.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.118+0000 7fe1b1006700 1 -- 172.21.15.6:0/2797516826 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1ac10e310 con 0x7fe1ac0fbe50 2024-02-09T21:58:04.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.119+0000 7fe1917fa700 1 -- 172.21.15.6:0/2797516826 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe18c0052b0 con 0x7fe1ac0fbe50 2024-02-09T21:58:04.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.120+0000 7fe1937fe700 1 -- 172.21.15.6:0/2797516826 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fe19c01aae0 con 0x7fe1ac0fbe50 2024-02-09T21:58:04.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.121+0000 7fe1937fe700 1 --2- 172.21.15.6:0/2797516826 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fe1940692a0 0x7fe19406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:58:04.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.121+0000 7fe1937fe700 1 -- 172.21.15.6:0/2797516826 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fe19c08aa90 con 0x7fe1ac0fbe50 2024-02-09T21:58:04.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.121+0000 7fe1aa59c700 1 --2- 172.21.15.6:0/2797516826 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fe1940692a0 0x7fe19406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:58:04.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.122+0000 7fe1aa59c700 1 --2- 172.21.15.6:0/2797516826 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fe1940692a0 0x7fe19406b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0009fd0 tx=0x7fe1a000b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:58:04.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.129+0000 7fe1937fe700 1 -- 172.21.15.6:0/2797516826 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe19c05a850 con 0x7fe1ac0fbe50 2024-02-09T21:58:04.194 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:03 smithi006 ceph-mon[29258]: pgmap v216: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 5 op/s 2024-02-09T21:58:04.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:03 smithi090 ceph-mon[36996]: pgmap v216: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 5 op/s 2024-02-09T21:58:04.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.437+0000 7fe1917fa700 1 -- 172.21.15.6:0/2797516826 --> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe18c000bc0 con 0x7fe1940692a0 2024-02-09T21:58:04.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.438+0000 7fe1937fe700 1 -- 172.21.15.6:0/2797516826 <== mgr.14216 v2:172.21.15.6:6800/2624137529 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7fe18c000bc0 con 0x7fe1940692a0 2024-02-09T21:58:04.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.441+0000 7fe1917fa700 1 -- 172.21.15.6:0/2797516826 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fe1940692a0 msgr2=0x7fe19406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:58:04.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.441+0000 7fe1917fa700 1 --2- 172.21.15.6:0/2797516826 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fe1940692a0 0x7fe19406b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0009fd0 tx=0x7fe1a000b040 comp rx=0 tx=0).stop 2024-02-09T21:58:04.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.441+0000 7fe1917fa700 1 -- 172.21.15.6:0/2797516826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1ac0fbe50 msgr2=0x7fe1ac10c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:58:04.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.441+0000 7fe1917fa700 1 --2- 172.21.15.6:0/2797516826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1ac0fbe50 0x7fe1ac10c7d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe19c009fd0 tx=0x7fe19c009e30 comp rx=0 tx=0).stop 2024-02-09T21:58:04.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.441+0000 7fe1917fa700 1 -- 172.21.15.6:0/2797516826 shutdown_connections 2024-02-09T21:58:04.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.441+0000 7fe1917fa700 1 --2- 172.21.15.6:0/2797516826 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fe1940692a0 0x7fe19406b750 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:04.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.441+0000 7fe1917fa700 1 --2- 172.21.15.6:0/2797516826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1ac0fbe50 0x7fe1ac10c7d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:04.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.441+0000 7fe1917fa700 1 --2- 172.21.15.6:0/2797516826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1ac0fcfc0 0x7fe1ac10cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:04.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.441+0000 7fe1917fa700 1 -- 172.21.15.6:0/2797516826 >> 172.21.15.6:0/2797516826 conn(0x7fe1ac0f7510 msgr2=0x7fe1ac100280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:58:04.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.441+0000 7fe1917fa700 1 -- 172.21.15.6:0/2797516826 shutdown_connections 2024-02-09T21:58:04.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.441+0000 7fe1917fa700 1 -- 172.21.15.6:0/2797516826 wait complete. 2024-02-09T21:58:04.450 INFO:tasks.workunit.client.1.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-02-09T21:58:04.461 INFO:teuthology.orchestra.run.smithi006.stdout:true 2024-02-09T21:58:04.549 INFO:tasks.workunit.client.1.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-02-09T21:58:04.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.554+0000 7fd313929700 1 -- 172.21.15.6:0/2169707123 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd30c1000c0 msgr2=0x7fd30c102520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:58:04.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.554+0000 7fd313929700 1 --2- 172.21.15.6:0/2169707123 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd30c1000c0 0x7fd30c102520 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc004940 tx=0x7fd2fc009e30 comp rx=0 tx=0).stop 2024-02-09T21:58:04.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.554+0000 7fd313929700 1 -- 172.21.15.6:0/2169707123 shutdown_connections 2024-02-09T21:58:04.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.554+0000 7fd313929700 1 --2- 172.21.15.6:0/2169707123 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd30c1000c0 0x7fd30c102520 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:04.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.554+0000 7fd313929700 1 --2- 172.21.15.6:0/2169707123 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd30c066c70 0x7fd30c0ffaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:04.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.554+0000 7fd313929700 1 -- 172.21.15.6:0/2169707123 >> 172.21.15.6:0/2169707123 conn(0x7fd30c0f7500 msgr2=0x7fd30c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:58:04.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.554+0000 7fd313929700 1 -- 172.21.15.6:0/2169707123 shutdown_connections 2024-02-09T21:58:04.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.554+0000 7fd313929700 1 -- 172.21.15.6:0/2169707123 wait complete. 2024-02-09T21:58:04.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.555+0000 7fd313929700 1 Processor -- start 2024-02-09T21:58:04.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.555+0000 7fd313929700 1 -- start start 2024-02-09T21:58:04.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.555+0000 7fd313929700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd30c066c70 0x7fd30c0fd570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:58:04.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.555+0000 7fd313929700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd30c0fb9f0 0x7fd30c0fbe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:58:04.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.555+0000 7fd313929700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd30c111730 con 0x7fd30c0fb9f0 2024-02-09T21:58:04.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.555+0000 7fd313929700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd30c111870 con 0x7fd30c066c70 2024-02-09T21:58:04.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.555+0000 7fd310ec4700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd30c0fb9f0 0x7fd30c0fbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:58:04.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.555+0000 7fd310ec4700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd30c0fb9f0 0x7fd30c0fbe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39462/0 (socket says 172.21.15.6:39462) 2024-02-09T21:58:04.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.555+0000 7fd310ec4700 1 -- 172.21.15.6:0/2501386978 learned_addr learned my addr 172.21.15.6:0/2501386978 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:58:04.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.555+0000 7fd3116c5700 1 --2- 172.21.15.6:0/2501386978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd30c066c70 0x7fd30c0fd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:58:04.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.556+0000 7fd310ec4700 1 -- 172.21.15.6:0/2501386978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd30c066c70 msgr2=0x7fd30c0fd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:58:04.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.556+0000 7fd310ec4700 1 --2- 172.21.15.6:0/2501386978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd30c066c70 0x7fd30c0fd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:04.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.556+0000 7fd310ec4700 1 -- 172.21.15.6:0/2501386978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2fc005040 con 0x7fd30c0fb9f0 2024-02-09T21:58:04.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.556+0000 7fd310ec4700 1 --2- 172.21.15.6:0/2501386978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd30c0fb9f0 0x7fd30c0fbe60 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc009fd0 tx=0x7fd2fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:58:04.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.556+0000 7fd301ffb700 1 -- 172.21.15.6:0/2501386978 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2fc00e040 con 0x7fd30c0fb9f0 2024-02-09T21:58:04.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.556+0000 7fd313929700 1 -- 172.21.15.6:0/2501386978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd30c0fc400 con 0x7fd30c0fb9f0 2024-02-09T21:58:04.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.556+0000 7fd301ffb700 1 -- 172.21.15.6:0/2501386978 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd2fc00bbf0 con 0x7fd30c0fb9f0 2024-02-09T21:58:04.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.556+0000 7fd301ffb700 1 -- 172.21.15.6:0/2501386978 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2fc025400 con 0x7fd30c0fb9f0 2024-02-09T21:58:04.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.556+0000 7fd313929700 1 -- 172.21.15.6:0/2501386978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd30c11a2e0 con 0x7fd30c0fb9f0 2024-02-09T21:58:04.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.557+0000 7fd313929700 1 -- 172.21.15.6:0/2501386978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd30c107a50 con 0x7fd30c0fb9f0 2024-02-09T21:58:04.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.558+0000 7fd301ffb700 1 -- 172.21.15.6:0/2501386978 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fd2fc01c3f0 con 0x7fd30c0fb9f0 2024-02-09T21:58:04.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.558+0000 7fd301ffb700 1 --2- 172.21.15.6:0/2501386978 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fd2f80692a0 0x7fd2f806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:58:04.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.559+0000 7fd301ffb700 1 -- 172.21.15.6:0/2501386978 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fd2fc088470 con 0x7fd30c0fb9f0 2024-02-09T21:58:04.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.559+0000 7fd3116c5700 1 --2- 172.21.15.6:0/2501386978 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fd2f80692a0 0x7fd2f806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:58:04.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.559+0000 7fd3116c5700 1 --2- 172.21.15.6:0/2501386978 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fd2f80692a0 0x7fd2f806b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd30c0fd300 tx=0x7fd308009ee0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:58:04.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.563+0000 7fd301ffb700 1 -- 172.21.15.6:0/2501386978 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd2fc0581b0 con 0x7fd30c0fb9f0 2024-02-09T21:58:04.591 INFO:tasks.workunit.client.1.smithi090.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-02-09T21:58:04.592 INFO:tasks.workunit.client.1.smithi090.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-02-09T21:58:04.592 INFO:tasks.workunit.client.1.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-02-09T21:58:04.632 INFO:tasks.workunit.client.1.smithi090.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-02-09T21:58:04.635 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-09T21:58:04.635 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-02-09T21:58:04.650 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-02-09T21:58:04.653 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-02-09T21:58:04.653 DEBUG:teuthology.orchestra.run.smithi090:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-02-09T21:58:04.712 INFO:tasks.workunit.client.1.smithi090.stderr:+ mkdir -p fsstress 2024-02-09T21:58:04.718 INFO:tasks.workunit.client.1.smithi090.stderr:+ pushd fsstress 2024-02-09T21:58:04.719 INFO:tasks.workunit.client.1.smithi090.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-02-09T21:58:04.719 INFO:tasks.workunit.client.1.smithi090.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-02-09T21:58:04.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.969+0000 7fd313929700 1 -- 172.21.15.6:0/2501386978 --> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd30c02b310 con 0x7fd2f80692a0 2024-02-09T21:58:04.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.970+0000 7fd301ffb700 1 -- 172.21.15.6:0/2501386978 <== mgr.14216 v2:172.21.15.6:6800/2624137529 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7fd30c02b310 con 0x7fd2f80692a0 2024-02-09T21:58:04.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.973+0000 7fd313929700 1 -- 172.21.15.6:0/2501386978 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fd2f80692a0 msgr2=0x7fd2f806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:58:04.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.973+0000 7fd313929700 1 --2- 172.21.15.6:0/2501386978 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fd2f80692a0 0x7fd2f806b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd30c0fd300 tx=0x7fd308009ee0 comp rx=0 tx=0).stop 2024-02-09T21:58:04.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.973+0000 7fd313929700 1 -- 172.21.15.6:0/2501386978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd30c0fb9f0 msgr2=0x7fd30c0fbe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:58:04.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.973+0000 7fd313929700 1 --2- 172.21.15.6:0/2501386978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd30c0fb9f0 0x7fd30c0fbe60 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc009fd0 tx=0x7fd2fc009e30 comp rx=0 tx=0).stop 2024-02-09T21:58:04.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.973+0000 7fd313929700 1 -- 172.21.15.6:0/2501386978 shutdown_connections 2024-02-09T21:58:04.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.973+0000 7fd313929700 1 --2- 172.21.15.6:0/2501386978 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fd2f80692a0 0x7fd2f806b750 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:04.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.973+0000 7fd313929700 1 --2- 172.21.15.6:0/2501386978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd30c066c70 0x7fd30c0fd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:04.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.973+0000 7fd313929700 1 --2- 172.21.15.6:0/2501386978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd30c0fb9f0 0x7fd30c0fbe60 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:04.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.973+0000 7fd313929700 1 -- 172.21.15.6:0/2501386978 >> 172.21.15.6:0/2501386978 conn(0x7fd30c0f7500 msgr2=0x7fd30c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:58:04.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.974+0000 7fd313929700 1 -- 172.21.15.6:0/2501386978 shutdown_connections 2024-02-09T21:58:04.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:04.974+0000 7fd313929700 1 -- 172.21.15.6:0/2501386978 wait complete. 2024-02-09T21:58:05.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.091+0000 7fd4cd74f700 1 -- 172.21.15.6:0/2050112535 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4c80fbe40 msgr2=0x7fd4c80fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:58:05.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.091+0000 7fd4cd74f700 1 --2- 172.21.15.6:0/2050112535 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4c80fbe40 0x7fd4c80fc250 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd4b8004940 tx=0x7fd4b8009e30 comp rx=0 tx=0).stop 2024-02-09T21:58:05.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.091+0000 7fd4cd74f700 1 -- 172.21.15.6:0/2050112535 shutdown_connections 2024-02-09T21:58:05.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.091+0000 7fd4cd74f700 1 --2- 172.21.15.6:0/2050112535 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4c80fcfb0 0x7fd4c80fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:05.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.091+0000 7fd4cd74f700 1 --2- 172.21.15.6:0/2050112535 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4c80fbe40 0x7fd4c80fc250 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:05.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.091+0000 7fd4cd74f700 1 -- 172.21.15.6:0/2050112535 >> 172.21.15.6:0/2050112535 conn(0x7fd4c80f74e0 msgr2=0x7fd4c80f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:58:05.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.091+0000 7fd4cd74f700 1 -- 172.21.15.6:0/2050112535 shutdown_connections 2024-02-09T21:58:05.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.091+0000 7fd4cd74f700 1 -- 172.21.15.6:0/2050112535 wait complete. 2024-02-09T21:58:05.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.092+0000 7fd4cd74f700 1 Processor -- start 2024-02-09T21:58:05.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.092+0000 7fd4cd74f700 1 -- start start 2024-02-09T21:58:05.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.093+0000 7fd4cd74f700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4c80fbe40 0x7fd4c810e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:58:05.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.093+0000 7fd4cd74f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4c80fcfb0 0x7fd4c8110940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:58:05.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.093+0000 7fd4cd74f700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c80664c0 con 0x7fd4c80fbe40 2024-02-09T21:58:05.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.093+0000 7fd4cd74f700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c8066600 con 0x7fd4c80fcfb0 2024-02-09T21:58:05.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.093+0000 7fd4c6ffd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4c80fbe40 0x7fd4c810e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:58:05.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.093+0000 7fd4c6ffd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4c80fbe40 0x7fd4c810e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39486/0 (socket says 172.21.15.6:39486) 2024-02-09T21:58:05.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.093+0000 7fd4c6ffd700 1 -- 172.21.15.6:0/264816011 learned_addr learned my addr 172.21.15.6:0/264816011 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:58:05.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.093+0000 7fd4c67fc700 1 --2- 172.21.15.6:0/264816011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4c80fcfb0 0x7fd4c8110940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:58:05.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.093+0000 7fd4c67fc700 1 -- 172.21.15.6:0/264816011 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4c80fbe40 msgr2=0x7fd4c810e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:58:05.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.093+0000 7fd4c67fc700 1 --2- 172.21.15.6:0/264816011 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4c80fbe40 0x7fd4c810e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:05.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.093+0000 7fd4c67fc700 1 -- 172.21.15.6:0/264816011 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b8005040 con 0x7fd4c80fcfb0 2024-02-09T21:58:05.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.093+0000 7fd4c6ffd700 1 --2- 172.21.15.6:0/264816011 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4c80fbe40 0x7fd4c810e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T21:58:05.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.093+0000 7fd4c67fc700 1 --2- 172.21.15.6:0/264816011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4c80fcfb0 0x7fd4c8110940 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc002820 tx=0x7fd4bc004570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:58:05.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.094+0000 7fd4af7fe700 1 -- 172.21.15.6:0/264816011 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4bc007cc0 con 0x7fd4c80fcfb0 2024-02-09T21:58:05.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.094+0000 7fd4af7fe700 1 -- 172.21.15.6:0/264816011 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd4bc00f040 con 0x7fd4c80fcfb0 2024-02-09T21:58:05.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.094+0000 7fd4af7fe700 1 -- 172.21.15.6:0/264816011 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4bc013890 con 0x7fd4c80fcfb0 2024-02-09T21:58:05.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.094+0000 7fd4cd74f700 1 -- 172.21.15.6:0/264816011 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4c8110fa0 con 0x7fd4c80fcfb0 2024-02-09T21:58:05.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.094+0000 7fd4cd74f700 1 -- 172.21.15.6:0/264816011 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4c81114e0 con 0x7fd4c80fcfb0 2024-02-09T21:58:05.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.095+0000 7fd4cd74f700 1 -- 172.21.15.6:0/264816011 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4c804bed0 con 0x7fd4c80fcfb0 2024-02-09T21:58:05.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.095+0000 7fd4af7fe700 1 -- 172.21.15.6:0/264816011 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fd4bc009070 con 0x7fd4c80fcfb0 2024-02-09T21:58:05.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.096+0000 7fd4af7fe700 1 --2- 172.21.15.6:0/264816011 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fd4b00692a0 0x7fd4b006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:58:05.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.096+0000 7fd4af7fe700 1 -- 172.21.15.6:0/264816011 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fd4bc086e50 con 0x7fd4c80fcfb0 2024-02-09T21:58:05.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.096+0000 7fd4c6ffd700 1 --2- 172.21.15.6:0/264816011 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fd4b00692a0 0x7fd4b006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:58:05.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.096+0000 7fd4c6ffd700 1 --2- 172.21.15.6:0/264816011 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fd4b00692a0 0x7fd4b006b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd4b800b3b0 tx=0x7fd4b8009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:58:05.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.100+0000 7fd4af7fe700 1 -- 172.21.15.6:0/264816011 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd4bc0530e0 con 0x7fd4c80fcfb0 2024-02-09T21:58:05.366 INFO:tasks.workunit.client.1.smithi090.stderr:+ tar xzf ltp-full.tgz 2024-02-09T21:58:05.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.387+0000 7fd4cd74f700 1 -- 172.21.15.6:0/264816011 --> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd4c8101990 con 0x7fd4b00692a0 2024-02-09T21:58:05.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.405+0000 7fd4af7fe700 1 -- 172.21.15.6:0/264816011 <== mgr.14216 v2:172.21.15.6:6800/2624137529 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7fd4c8101990 con 0x7fd4b00692a0 2024-02-09T21:58:05.409 INFO:teuthology.orchestra.run.smithi006.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-09T21:58:05.410 INFO:teuthology.orchestra.run.smithi006.stdout:alertmanager.smithi006 smithi006 *:9093,9094 running (5m) 57s ago 6m 25.5M - 0.25.0 c8568f914cd2 5c39f25f512b 2024-02-09T21:58:05.410 INFO:teuthology.orchestra.run.smithi006.stdout:crash.smithi006 smithi006 running (7m) 57s ago 7m 6665k - 17.2.7-505-g72ce7c1d a2a0bd126c96 821e62ef6577 2024-02-09T21:58:05.410 INFO:teuthology.orchestra.run.smithi006.stdout:crash.smithi090 smithi090 running (5m) 64s ago 5m 6665k - 17.2.7-505-g72ce7c1d a2a0bd126c96 c9a50a0d16b7 2024-02-09T21:58:05.410 INFO:teuthology.orchestra.run.smithi006.stdout:grafana.smithi006 smithi006 *:3000 running (5m) 57s ago 6m 87.6M - 9.4.7 954c08fa6188 32ff2ec358e5 2024-02-09T21:58:05.410 INFO:teuthology.orchestra.run.smithi006.stdout:mds.cephfs.smithi006.qyamnu smithi006 running (77s) 57s ago 77s 13.5M - 17.2.7-505-g72ce7c1d a2a0bd126c96 4c9a0d2dc5b1 2024-02-09T21:58:05.410 INFO:teuthology.orchestra.run.smithi006.stdout:mds.cephfs.smithi006.ypgiqs smithi006 running (83s) 57s ago 82s 14.8M - 17.2.7-505-g72ce7c1d a2a0bd126c96 ce481c5f5b05 2024-02-09T21:58:05.410 INFO:teuthology.orchestra.run.smithi006.stdout:mds.cephfs.smithi090.cgsuda smithi090 running (80s) 64s ago 80s 16.7M - 17.2.7-505-g72ce7c1d a2a0bd126c96 2cd6bcefa660 2024-02-09T21:58:05.410 INFO:teuthology.orchestra.run.smithi006.stdout:mds.cephfs.smithi090.mipeaa smithi090 running (75s) 64s ago 74s 15.5M - 17.2.7-505-g72ce7c1d a2a0bd126c96 80d841dc4e6f 2024-02-09T21:58:05.410 INFO:teuthology.orchestra.run.smithi006.stdout:mgr.smithi006.glirxt smithi006 *:9283 running (8m) 57s ago 8m 483M - 17.2.7-505-g72ce7c1d a2a0bd126c96 133e424caf5e 2024-02-09T21:58:05.410 INFO:teuthology.orchestra.run.smithi006.stdout:mgr.smithi090.csxosj smithi090 *:8443,9283 running (5m) 64s ago 5m 434M - 17.2.7-505-g72ce7c1d a2a0bd126c96 1c0eaba0034b 2024-02-09T21:58:05.410 INFO:teuthology.orchestra.run.smithi006.stdout:mon.smithi006 smithi006 running (8m) 57s ago 8m 46.6M 2048M 17.2.7-505-g72ce7c1d a2a0bd126c96 f8cba7495eab 2024-02-09T21:58:05.410 INFO:teuthology.orchestra.run.smithi006.stdout:mon.smithi090 smithi090 running (5m) 64s ago 5m 39.0M 2048M 17.2.7-505-g72ce7c1d a2a0bd126c96 4e4911a241f4 2024-02-09T21:58:05.410 INFO:teuthology.orchestra.run.smithi006.stdout:node-exporter.smithi006 smithi006 *:9100 running (7m) 57s ago 7m 16.0M - 1.5.0 0da6a335fe13 8c411b417300 2024-02-09T21:58:05.410 INFO:teuthology.orchestra.run.smithi006.stdout:node-exporter.smithi090 smithi090 *:9100 running (5m) 64s ago 5m 16.2M - 1.5.0 0da6a335fe13 274157586c39 2024-02-09T21:58:05.411 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0 smithi006 running (5m) 57s ago 5m 48.6M 4289M 17.2.7-505-g72ce7c1d a2a0bd126c96 1aea34e55f65 2024-02-09T21:58:05.411 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1 smithi006 running (4m) 57s ago 4m 47.2M 4289M 17.2.7-505-g72ce7c1d a2a0bd126c96 58f54ee70423 2024-02-09T21:58:05.411 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2 smithi006 running (4m) 57s ago 4m 44.5M 4289M 17.2.7-505-g72ce7c1d a2a0bd126c96 9dfe76e205a6 2024-02-09T21:58:05.411 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3 smithi090 running (3m) 64s ago 3m 47.3M 5312M 17.2.7-505-g72ce7c1d a2a0bd126c96 7cadb80e5790 2024-02-09T21:58:05.411 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4 smithi090 running (3m) 64s ago 3m 41.9M 5312M 17.2.7-505-g72ce7c1d a2a0bd126c96 1e173e0c2c75 2024-02-09T21:58:05.411 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5 smithi090 running (2m) 64s ago 2m 44.3M 5312M 17.2.7-505-g72ce7c1d a2a0bd126c96 bc873d1a2623 2024-02-09T21:58:05.411 INFO:teuthology.orchestra.run.smithi006.stdout:prometheus.smithi006 smithi006 *:9095 running (5m) 57s ago 6m 55.6M - 2.43.0 a07b618ecd1d ac6cb4bbb2d2 2024-02-09T21:58:05.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.408+0000 7fd4cd74f700 1 -- 172.21.15.6:0/264816011 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fd4b00692a0 msgr2=0x7fd4b006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:58:05.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.408+0000 7fd4cd74f700 1 --2- 172.21.15.6:0/264816011 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fd4b00692a0 0x7fd4b006b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd4b800b3b0 tx=0x7fd4b8009e30 comp rx=0 tx=0).stop 2024-02-09T21:58:05.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.408+0000 7fd4cd74f700 1 -- 172.21.15.6:0/264816011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4c80fcfb0 msgr2=0x7fd4c8110940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:58:05.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.408+0000 7fd4cd74f700 1 --2- 172.21.15.6:0/264816011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4c80fcfb0 0x7fd4c8110940 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc002820 tx=0x7fd4bc004570 comp rx=0 tx=0).stop 2024-02-09T21:58:05.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.408+0000 7fd4cd74f700 1 -- 172.21.15.6:0/264816011 shutdown_connections 2024-02-09T21:58:05.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.408+0000 7fd4cd74f700 1 --2- 172.21.15.6:0/264816011 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fd4b00692a0 0x7fd4b006b750 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:05.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.408+0000 7fd4cd74f700 1 --2- 172.21.15.6:0/264816011 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4c80fbe40 0x7fd4c810e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:05.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.408+0000 7fd4cd74f700 1 --2- 172.21.15.6:0/264816011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4c80fcfb0 0x7fd4c8110940 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:05.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.408+0000 7fd4cd74f700 1 -- 172.21.15.6:0/264816011 >> 172.21.15.6:0/264816011 conn(0x7fd4c80f74e0 msgr2=0x7fd4c8100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:58:05.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.409+0000 7fd4cd74f700 1 -- 172.21.15.6:0/264816011 shutdown_connections 2024-02-09T21:58:05.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.409+0000 7fd4cd74f700 1 -- 172.21.15.6:0/264816011 wait complete. 2024-02-09T21:58:05.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.571+0000 7f26ce7be700 1 -- 172.21.15.6:0/4269476364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26c8073af0 msgr2=0x7f26c8073f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:58:05.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.571+0000 7f26ce7be700 1 --2- 172.21.15.6:0/4269476364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26c8073af0 0x7f26c8073f60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f26b8004940 tx=0x7f26b8009e30 comp rx=0 tx=0).stop 2024-02-09T21:58:05.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.572+0000 7f26ce7be700 1 -- 172.21.15.6:0/4269476364 shutdown_connections 2024-02-09T21:58:05.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.572+0000 7f26ce7be700 1 --2- 172.21.15.6:0/4269476364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26c8073af0 0x7f26c8073f60 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:05.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.572+0000 7f26ce7be700 1 --2- 172.21.15.6:0/4269476364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26c8072890 0x7f26c8072ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:05.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.572+0000 7f26ce7be700 1 -- 172.21.15.6:0/4269476364 >> 172.21.15.6:0/4269476364 conn(0x7f26c80fa920 msgr2=0x7f26c80fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:58:05.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.572+0000 7f26ce7be700 1 -- 172.21.15.6:0/4269476364 shutdown_connections 2024-02-09T21:58:05.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.572+0000 7f26ce7be700 1 -- 172.21.15.6:0/4269476364 wait complete. 2024-02-09T21:58:05.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.572+0000 7f26ce7be700 1 Processor -- start 2024-02-09T21:58:05.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.573+0000 7f26ce7be700 1 -- start start 2024-02-09T21:58:05.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.573+0000 7f26ce7be700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26c8072890 0x7f26c806efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:58:05.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.573+0000 7f26ce7be700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26c8073af0 0x7f26c806f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:58:05.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.573+0000 7f26ce7be700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c806fb50 con 0x7f26c8073af0 2024-02-09T21:58:05.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.573+0000 7f26ce7be700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c811db60 con 0x7f26c8072890 2024-02-09T21:58:05.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.573+0000 7f26c7fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26c8072890 0x7f26c806efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:58:05.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.573+0000 7f26c77fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26c8073af0 0x7f26c806f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:58:05.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.573+0000 7f26c7fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26c8072890 0x7f26c806efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.6:47136/0 (socket says 172.21.15.6:47136) 2024-02-09T21:58:05.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.573+0000 7f26c7fff700 1 -- 172.21.15.6:0/1148074417 learned_addr learned my addr 172.21.15.6:0/1148074417 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:58:05.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.573+0000 7f26c7fff700 1 -- 172.21.15.6:0/1148074417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26c8073af0 msgr2=0x7f26c806f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:58:05.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.573+0000 7f26c7fff700 1 --2- 172.21.15.6:0/1148074417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26c8073af0 0x7f26c806f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:05.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.573+0000 7f26c7fff700 1 -- 172.21.15.6:0/1148074417 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b8005040 con 0x7f26c8072890 2024-02-09T21:58:05.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.574+0000 7f26c7fff700 1 --2- 172.21.15.6:0/1148074417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26c8072890 0x7f26c806efc0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f26b000eb40 tx=0x7f26b000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:58:05.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.574+0000 7f26c4ff9700 1 -- 172.21.15.6:0/1148074417 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26b0007cc0 con 0x7f26c8072890 2024-02-09T21:58:05.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.574+0000 7f26c4ff9700 1 -- 172.21.15.6:0/1148074417 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f26b000f040 con 0x7f26c8072890 2024-02-09T21:58:05.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.574+0000 7f26c4ff9700 1 -- 172.21.15.6:0/1148074417 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26b000ca10 con 0x7f26c8072890 2024-02-09T21:58:05.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.574+0000 7f26ce7be700 1 -- 172.21.15.6:0/1148074417 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26c811dde0 con 0x7f26c8072890 2024-02-09T21:58:05.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.574+0000 7f26ce7be700 1 -- 172.21.15.6:0/1148074417 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26c811e320 con 0x7f26c8072890 2024-02-09T21:58:05.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.575+0000 7f26ce7be700 1 -- 172.21.15.6:0/1148074417 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26c804bed0 con 0x7f26c8072890 2024-02-09T21:58:05.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.576+0000 7f26c4ff9700 1 -- 172.21.15.6:0/1148074417 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f26b000cb70 con 0x7f26c8072890 2024-02-09T21:58:05.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.577+0000 7f26c4ff9700 1 --2- 172.21.15.6:0/1148074417 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f26b40692b0 0x7f26b406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:58:05.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.577+0000 7f26c4ff9700 1 -- 172.21.15.6:0/1148074417 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f26b0009600 con 0x7f26c8072890 2024-02-09T21:58:05.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.577+0000 7f26c77fe700 1 --2- 172.21.15.6:0/1148074417 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f26b40692b0 0x7f26b406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:58:05.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.578+0000 7f26c77fe700 1 --2- 172.21.15.6:0/1148074417 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f26b40692b0 0x7f26b406b760 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f26b800b0d0 tx=0x7f26b8009e30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:58:05.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:05.582+0000 7f26c4ff9700 1 -- 172.21.15.6:0/1148074417 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f26b0053800 con 0x7f26c8072890 2024-02-09T21:58:06.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:05 smithi006 ceph-mon[29258]: from='client.14546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:58:06.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:05 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:58:06.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:05 smithi006 ceph-mon[29258]: Upgrade: Target is version 19.0.0-1361-g290212a1 (unknown) 2024-02-09T21:58:06.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:05 smithi006 ceph-mon[29258]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-09T21:58:06.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:05 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:58:06.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:05 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:58:06.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:05 smithi006 ceph-mon[29258]: Upgrade: Need to upgrade myself (mgr.smithi006.glirxt) 2024-02-09T21:58:06.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:05 smithi006 ceph-mon[29258]: pgmap v217: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 170 B/s wr, 4 op/s 2024-02-09T21:58:06.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:05 smithi006 ceph-mon[29258]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:58:06.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:05 smithi006 ceph-mon[29258]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 on smithi090 2024-02-09T21:58:06.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.159+0000 7f26ce7be700 1 -- 172.21.15.6:0/1148074417 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f26c811df70 con 0x7f26c8072890 2024-02-09T21:58:06.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.161+0000 7f26c4ff9700 1 -- 172.21.15.6:0/1148074417 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f26b0053800 con 0x7f26c8072890 2024-02-09T21:58:06.165 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-02-09T21:58:06.165 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": { 2024-02-09T21:58:06.165 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)": 2 2024-02-09T21:58:06.165 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-02-09T21:58:06.165 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": { 2024-02-09T21:58:06.165 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)": 2 2024-02-09T21:58:06.165 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-02-09T21:58:06.165 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": { 2024-02-09T21:58:06.165 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)": 6 2024-02-09T21:58:06.165 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-02-09T21:58:06.166 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": { 2024-02-09T21:58:06.166 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)": 4 2024-02-09T21:58:06.166 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-02-09T21:58:06.166 INFO:teuthology.orchestra.run.smithi006.stdout: "overall": { 2024-02-09T21:58:06.166 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph version 17.2.7-505-g72ce7c1d (72ce7c1d1d5a2a1807db94c08aa742a26968c21f) quincy (stable)": 14 2024-02-09T21:58:06.166 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-02-09T21:58:06.166 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-02-09T21:58:06.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.164+0000 7f26ce7be700 1 -- 172.21.15.6:0/1148074417 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f26b40692b0 msgr2=0x7f26b406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:58:06.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.164+0000 7f26ce7be700 1 --2- 172.21.15.6:0/1148074417 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f26b40692b0 0x7f26b406b760 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f26b800b0d0 tx=0x7f26b8009e30 comp rx=0 tx=0).stop 2024-02-09T21:58:06.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.164+0000 7f26ce7be700 1 -- 172.21.15.6:0/1148074417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26c8072890 msgr2=0x7f26c806efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:58:06.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.164+0000 7f26ce7be700 1 --2- 172.21.15.6:0/1148074417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26c8072890 0x7f26c806efc0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f26b000eb40 tx=0x7f26b000b040 comp rx=0 tx=0).stop 2024-02-09T21:58:06.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.165+0000 7f26ce7be700 1 -- 172.21.15.6:0/1148074417 shutdown_connections 2024-02-09T21:58:06.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.165+0000 7f26ce7be700 1 --2- 172.21.15.6:0/1148074417 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7f26b40692b0 0x7f26b406b760 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:06.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.165+0000 7f26ce7be700 1 --2- 172.21.15.6:0/1148074417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26c8072890 0x7f26c806efc0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:06.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.165+0000 7f26ce7be700 1 --2- 172.21.15.6:0/1148074417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26c8073af0 0x7f26c806f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:06.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.165+0000 7f26ce7be700 1 -- 172.21.15.6:0/1148074417 >> 172.21.15.6:0/1148074417 conn(0x7f26c80fa920 msgr2=0x7f26c8109460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:58:06.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.165+0000 7f26ce7be700 1 -- 172.21.15.6:0/1148074417 shutdown_connections 2024-02-09T21:58:06.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.165+0000 7f26ce7be700 1 -- 172.21.15.6:0/1148074417 wait complete. 2024-02-09T21:58:06.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:05 smithi090 ceph-mon[36996]: from='client.14546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:58:06.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:05 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:58:06.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:05 smithi090 ceph-mon[36996]: Upgrade: Target is version 19.0.0-1361-g290212a1 (unknown) 2024-02-09T21:58:06.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:05 smithi090 ceph-mon[36996]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-09T21:58:06.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:05 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T21:58:06.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:05 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' 2024-02-09T21:58:06.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:05 smithi090 ceph-mon[36996]: Upgrade: Need to upgrade myself (mgr.smithi006.glirxt) 2024-02-09T21:58:06.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:05 smithi090 ceph-mon[36996]: pgmap v217: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 170 B/s wr, 4 op/s 2024-02-09T21:58:06.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:05 smithi090 ceph-mon[36996]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:58:06.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:05 smithi090 ceph-mon[36996]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 on smithi090 2024-02-09T21:58:06.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.331+0000 7fb06d1f3700 1 -- 172.21.15.6:0/4252305485 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0680fb5e0 msgr2=0x7fb0680fb9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:58:06.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.331+0000 7fb06d1f3700 1 --2- 172.21.15.6:0/4252305485 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0680fb5e0 0x7fb0680fb9f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb058004940 tx=0x7fb058009e30 comp rx=0 tx=0).stop 2024-02-09T21:58:06.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.332+0000 7fb06d1f3700 1 -- 172.21.15.6:0/4252305485 shutdown_connections 2024-02-09T21:58:06.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.332+0000 7fb06d1f3700 1 --2- 172.21.15.6:0/4252305485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0680fc7f0 0x7fb0680fcc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:06.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.332+0000 7fb06d1f3700 1 --2- 172.21.15.6:0/4252305485 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0680fb5e0 0x7fb0680fb9f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:06.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.332+0000 7fb06d1f3700 1 -- 172.21.15.6:0/4252305485 >> 172.21.15.6:0/4252305485 conn(0x7fb0680f6ca0 msgr2=0x7fb0680f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:58:06.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.332+0000 7fb06d1f3700 1 -- 172.21.15.6:0/4252305485 shutdown_connections 2024-02-09T21:58:06.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.332+0000 7fb06d1f3700 1 -- 172.21.15.6:0/4252305485 wait complete. 2024-02-09T21:58:06.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.333+0000 7fb06d1f3700 1 Processor -- start 2024-02-09T21:58:06.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.333+0000 7fb06d1f3700 1 -- start start 2024-02-09T21:58:06.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.334+0000 7fb06d1f3700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0680fb5e0 0x7fb06810b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:58:06.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.334+0000 7fb06d1f3700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0680fc7f0 0x7fb06810bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:58:06.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.334+0000 7fb06d1f3700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06810c2e0 con 0x7fb0680fb5e0 2024-02-09T21:58:06.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.334+0000 7fb06d1f3700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06810cb80 con 0x7fb0680fc7f0 2024-02-09T21:58:06.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.334+0000 7fb066d9d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0680fb5e0 0x7fb06810b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:58:06.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.334+0000 7fb066d9d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0680fb5e0 0x7fb06810b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39522/0 (socket says 172.21.15.6:39522) 2024-02-09T21:58:06.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.334+0000 7fb066d9d700 1 -- 172.21.15.6:0/1714370705 learned_addr learned my addr 172.21.15.6:0/1714370705 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-02-09T21:58:06.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.334+0000 7fb06659c700 1 --2- 172.21.15.6:0/1714370705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0680fc7f0 0x7fb06810bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:58:06.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.335+0000 7fb066d9d700 1 -- 172.21.15.6:0/1714370705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0680fc7f0 msgr2=0x7fb06810bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:58:06.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.335+0000 7fb066d9d700 1 --2- 172.21.15.6:0/1714370705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0680fc7f0 0x7fb06810bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:06.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.335+0000 7fb066d9d700 1 -- 172.21.15.6:0/1714370705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb05c005040 con 0x7fb0680fb5e0 2024-02-09T21:58:06.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.335+0000 7fb06659c700 1 --2- 172.21.15.6:0/1714370705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0680fc7f0 0x7fb06810bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T21:58:06.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.335+0000 7fb066d9d700 1 --2- 172.21.15.6:0/1714370705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0680fb5e0 0x7fb06810b7a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb058000c00 tx=0x7fb058009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:58:06.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.335+0000 7fb04f7fe700 1 -- 172.21.15.6:0/1714370705 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb058021070 con 0x7fb0680fb5e0 2024-02-09T21:58:06.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.335+0000 7fb04f7fe700 1 -- 172.21.15.6:0/1714370705 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb05801c400 con 0x7fb0680fb5e0 2024-02-09T21:58:06.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.335+0000 7fb06d1f3700 1 -- 172.21.15.6:0/1714370705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb058005040 con 0x7fb0680fb5e0 2024-02-09T21:58:06.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.336+0000 7fb06d1f3700 1 -- 172.21.15.6:0/1714370705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb06810d0f0 con 0x7fb0680fb5e0 2024-02-09T21:58:06.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.336+0000 7fb04f7fe700 1 -- 172.21.15.6:0/1714370705 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb05800e070 con 0x7fb0680fb5e0 2024-02-09T21:58:06.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.337+0000 7fb06d1f3700 1 -- 172.21.15.6:0/1714370705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb06804bed0 con 0x7fb0680fb5e0 2024-02-09T21:58:06.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.338+0000 7fb04f7fe700 1 -- 172.21.15.6:0/1714370705 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fb05801ca90 con 0x7fb0680fb5e0 2024-02-09T21:58:06.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.339+0000 7fb04f7fe700 1 --2- 172.21.15.6:0/1714370705 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fb050069310 0x7fb05006b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T21:58:06.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.339+0000 7fb04f7fe700 1 -- 172.21.15.6:0/1714370705 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fb058088cb0 con 0x7fb0680fb5e0 2024-02-09T21:58:06.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.340+0000 7fb06659c700 1 --2- 172.21.15.6:0/1714370705 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fb050069310 0x7fb05006b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T21:58:06.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.340+0000 7fb06659c700 1 --2- 172.21.15.6:0/1714370705 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fb050069310 0x7fb05006b7c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb05c00afd0 tx=0x7fb05c00c040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T21:58:06.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.345+0000 7fb04f7fe700 1 -- 172.21.15.6:0/1714370705 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb058058a50 con 0x7fb0680fb5e0 2024-02-09T21:58:06.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.763+0000 7fb06d1f3700 1 -- 172.21.15.6:0/1714370705 --> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb06810d600 con 0x7fb050069310 2024-02-09T21:58:06.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.765+0000 7fb04f7fe700 1 -- 172.21.15.6:0/1714370705 <== mgr.14216 v2:172.21.15.6:6800/2624137529 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7fb06810d600 con 0x7fb050069310 2024-02-09T21:58:06.769 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-02-09T21:58:06.769 INFO:teuthology.orchestra.run.smithi006.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-09T21:58:06.769 INFO:teuthology.orchestra.run.smithi006.stdout: "in_progress": true, 2024-02-09T21:58:06.769 INFO:teuthology.orchestra.run.smithi006.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-09T21:58:06.769 INFO:teuthology.orchestra.run.smithi006.stdout: "services_complete": [], 2024-02-09T21:58:06.769 INFO:teuthology.orchestra.run.smithi006.stdout: "progress": "0/2 daemons upgraded", 2024-02-09T21:58:06.769 INFO:teuthology.orchestra.run.smithi006.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 image on host smithi090", 2024-02-09T21:58:06.770 INFO:teuthology.orchestra.run.smithi006.stdout: "is_paused": false 2024-02-09T21:58:06.770 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-02-09T21:58:06.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.769+0000 7fb06d1f3700 1 -- 172.21.15.6:0/1714370705 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fb050069310 msgr2=0x7fb05006b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:58:06.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.769+0000 7fb06d1f3700 1 --2- 172.21.15.6:0/1714370705 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fb050069310 0x7fb05006b7c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb05c00afd0 tx=0x7fb05c00c040 comp rx=0 tx=0).stop 2024-02-09T21:58:06.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.769+0000 7fb06d1f3700 1 -- 172.21.15.6:0/1714370705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0680fb5e0 msgr2=0x7fb06810b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T21:58:06.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.769+0000 7fb06d1f3700 1 --2- 172.21.15.6:0/1714370705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0680fb5e0 0x7fb06810b7a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb058000c00 tx=0x7fb058009e30 comp rx=0 tx=0).stop 2024-02-09T21:58:06.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.769+0000 7fb06d1f3700 1 -- 172.21.15.6:0/1714370705 shutdown_connections 2024-02-09T21:58:06.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.769+0000 7fb06d1f3700 1 --2- 172.21.15.6:0/1714370705 >> [v2:172.21.15.6:6800/2624137529,v1:172.21.15.6:6801/2624137529] conn(0x7fb050069310 0x7fb05006b7c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:06.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.769+0000 7fb06d1f3700 1 --2- 172.21.15.6:0/1714370705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0680fb5e0 0x7fb06810b7a0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:06.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.769+0000 7fb06d1f3700 1 --2- 172.21.15.6:0/1714370705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0680fc7f0 0x7fb06810bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T21:58:06.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.769+0000 7fb06d1f3700 1 -- 172.21.15.6:0/1714370705 >> 172.21.15.6:0/1714370705 conn(0x7fb0680f6ca0 msgr2=0x7fb0680ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T21:58:06.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.770+0000 7fb06d1f3700 1 -- 172.21.15.6:0/1714370705 shutdown_connections 2024-02-09T21:58:06.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T21:58:06.770+0000 7fb06d1f3700 1 -- 172.21.15.6:0/1714370705 wait complete. 2024-02-09T21:58:07.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:06 smithi006 ceph-mon[29258]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:58:07.158 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:06 smithi006 ceph-mon[29258]: from='client.? 172.21.15.6:0/1148074417' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-09T21:58:07.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:06 smithi090 ceph-mon[36996]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:58:07.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:06 smithi090 ceph-mon[36996]: from='client.? 172.21.15.6:0/1148074417' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-09T21:58:08.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:07 smithi006 ceph-mon[29258]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:58:08.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:07 smithi006 ceph-mon[29258]: pgmap v218: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 4 op/s 2024-02-09T21:58:08.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:07 smithi090 ceph-mon[36996]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T21:58:08.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:07 smithi090 ceph-mon[36996]: pgmap v218: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 4 op/s 2024-02-09T21:58:10.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:09 smithi090 ceph-mon[36996]: pgmap v219: 65 pgs: 65 active+clean; 3.1 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 113 KiB/s rd, 217 KiB/s wr, 5 op/s 2024-02-09T21:58:10.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:09 smithi006 ceph-mon[29258]: pgmap v219: 65 pgs: 65 active+clean; 3.1 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 113 KiB/s rd, 217 KiB/s wr, 5 op/s 2024-02-09T21:58:12.102 INFO:tasks.workunit.client.0.smithi006.stderr:Updating files: 92% (11207/12104) Updating files: 93% (11257/12104) Updating files: 94% (11378/12104) Updating files: 95% (11499/12104) Updating files: 96% (11620/12104) Updating files: 97% (11741/12104) Updating files: 98% (11862/12104) Updating files: 99% (11983/12104) Updating files: 100% (12104/12104) Updating files: 100% (12104/12104), done. 2024-02-09T21:58:12.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:11 smithi090 ceph-mon[36996]: pgmap v220: 65 pgs: 65 active+clean; 3.1 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 112 KiB/s rd, 217 KiB/s wr, 4 op/s 2024-02-09T21:58:12.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:11 smithi090 ceph-mon[36996]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:58:12.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:11 smithi006 ceph-mon[29258]: pgmap v220: 65 pgs: 65 active+clean; 3.1 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 112 KiB/s rd, 217 KiB/s wr, 4 op/s 2024-02-09T21:58:12.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:11 smithi006 ceph-mon[29258]: from='mgr.14216 172.21.15.6:0/4103617958' entity='mgr.smithi006.glirxt' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-09T21:58:12.559 INFO:tasks.workunit.client.0.smithi006.stderr:Note: switching to '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'. 2024-02-09T21:58:12.559 INFO:tasks.workunit.client.0.smithi006.stderr: 2024-02-09T21:58:12.559 INFO:tasks.workunit.client.0.smithi006.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-09T21:58:12.559 INFO:tasks.workunit.client.0.smithi006.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-09T21:58:12.559 INFO:tasks.workunit.client.0.smithi006.stderr:state without impacting any branches by switching back to a branch. 2024-02-09T21:58:12.559 INFO:tasks.workunit.client.0.smithi006.stderr: 2024-02-09T21:58:12.559 INFO:tasks.workunit.client.0.smithi006.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-09T21:58:12.559 INFO:tasks.workunit.client.0.smithi006.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-09T21:58:12.559 INFO:tasks.workunit.client.0.smithi006.stderr: 2024-02-09T21:58:12.559 INFO:tasks.workunit.client.0.smithi006.stderr: git switch -c 2024-02-09T21:58:12.559 INFO:tasks.workunit.client.0.smithi006.stderr: 2024-02-09T21:58:12.560 INFO:tasks.workunit.client.0.smithi006.stderr:Or undo this operation with: 2024-02-09T21:58:12.560 INFO:tasks.workunit.client.0.smithi006.stderr: 2024-02-09T21:58:12.560 INFO:tasks.workunit.client.0.smithi006.stderr: git switch - 2024-02-09T21:58:12.560 INFO:tasks.workunit.client.0.smithi006.stderr: 2024-02-09T21:58:12.560 INFO:tasks.workunit.client.0.smithi006.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-09T21:58:12.560 INFO:tasks.workunit.client.0.smithi006.stderr: 2024-02-09T21:58:12.560 INFO:tasks.workunit.client.0.smithi006.stderr:HEAD is now at 290212a1bfd Merge pull request #53521 from xxhdx1985126/wip-62857 2024-02-09T21:58:12.563 DEBUG:teuthology.orchestra.run.smithi006:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-02-09T21:58:12.594 INFO:tasks.workunit.client.0.smithi006.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-02-09T21:58:12.596 INFO:tasks.workunit.client.0.smithi006.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-09T21:58:12.596 INFO:tasks.workunit.client.0.smithi006.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-02-09T21:58:13.458 INFO:tasks.workunit.client.0.smithi006.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-02-09T21:58:13.527 INFO:tasks.workunit.client.0.smithi006.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-02-09T21:58:13.567 INFO:tasks.workunit.client.0.smithi006.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-09T21:58:13.569 INFO:tasks.workunit.client.0.smithi006.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-09T21:58:13.569 INFO:tasks.workunit.client.0.smithi006.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-02-09T21:58:13.608 INFO:tasks.workunit.client.0.smithi006.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-09T21:58:13.610 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-02-09T21:58:13.611 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-02-09T21:58:13.666 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-02-09T21:58:13.667 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-02-09T21:58:13.667 DEBUG:teuthology.orchestra.run.smithi006:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-02-09T21:58:13.729 INFO:tasks.workunit.client.0.smithi006.stderr:+ mkdir -p fsstress 2024-02-09T21:58:13.732 INFO:tasks.workunit.client.0.smithi006.stderr:+ pushd fsstress 2024-02-09T21:58:13.732 INFO:tasks.workunit.client.0.smithi006.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-02-09T21:58:13.732 INFO:tasks.workunit.client.0.smithi006.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-02-09T21:58:14.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:13 smithi090 ceph-mon[36996]: pgmap v221: 65 pgs: 65 active+clean; 26 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 785 KiB/s rd, 2.2 MiB/s wr, 122 op/s 2024-02-09T21:58:14.205 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:13 smithi090 ceph-mon[36996]: fsmap cephfs:2 {0=cephfs.smithi006.ypgiqs=up:active,1=cephfs.smithi090.cgsuda=up:active} 2 up:standby-replay 2024-02-09T21:58:14.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:13 smithi006 ceph-mon[29258]: pgmap v221: 65 pgs: 65 active+clean; 26 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 785 KiB/s rd, 2.2 MiB/s wr, 122 op/s 2024-02-09T21:58:14.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:13 smithi006 ceph-mon[29258]: fsmap cephfs:2 {0=cephfs.smithi006.ypgiqs=up:active,1=cephfs.smithi090.cgsuda=up:active} 2 up:standby-replay 2024-02-09T21:58:14.603 INFO:tasks.workunit.client.0.smithi006.stderr:+ tar xzf ltp-full.tgz 2024-02-09T21:58:16.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:15 smithi006 ceph-mon[29258]: pgmap v222: 65 pgs: 65 active+clean; 49 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 911 KiB/s rd, 4.2 MiB/s wr, 348 op/s 2024-02-09T21:58:16.454 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:15 smithi090 ceph-mon[36996]: pgmap v222: 65 pgs: 65 active+clean; 49 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 911 KiB/s rd, 4.2 MiB/s wr, 348 op/s 2024-02-09T21:58:17.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:17 smithi006 ceph-mon[29258]: pgmap v223: 65 pgs: 65 active+clean; 50 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 997 KiB/s rd, 4.3 MiB/s wr, 349 op/s 2024-02-09T21:58:17.708 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:17 smithi090 ceph-mon[36996]: pgmap v223: 65 pgs: 65 active+clean; 50 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 997 KiB/s rd, 4.3 MiB/s wr, 349 op/s 2024-02-09T21:58:20.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:19 smithi090 ceph-mon[36996]: pgmap v224: 65 pgs: 65 active+clean; 57 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 4.9 MiB/s wr, 356 op/s 2024-02-09T21:58:20.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:19 smithi006 ceph-mon[29258]: pgmap v224: 65 pgs: 65 active+clean; 57 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 4.9 MiB/s wr, 356 op/s 2024-02-09T21:58:22.147 INFO:tasks.workunit.client.1.smithi090.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-02-09T21:58:22.147 INFO:tasks.workunit.client.1.smithi090.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-02-09T21:58:22.148 INFO:tasks.workunit.client.1.smithi090.stderr:+ make 2024-02-09T21:58:22.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 09 21:58:21 smithi090 ceph-mon[36996]: pgmap v225: 65 pgs: 65 active+clean; 62 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 5.1 MiB/s wr, 372 op/s 2024-02-09T21:58:22.218 INFO:tasks.workunit.client.1.smithi090.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-02-09T21:58:22.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 21:58:21 smithi006 ceph-mon[29258]: pgmap v225: 65 pgs: 65 active+clean; 62 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 5.1 MiB/s wr, 372 op/s 2024-02-09T21:58:45.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:45.096+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:45.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:45.096+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:45.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:45.096+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:46.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:46.088+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:46.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:46.088+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:46.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:46.088+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:46.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:45.903+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:46.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:45.903+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:46.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:45.903+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:47.084 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:46.859+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:47.085 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:46.859+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:47.085 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:46.859+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:47.085 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:46.726+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:47.085 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:46.726+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:47.085 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:46.726+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:47.403 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:47.080+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:47.403 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:47.080+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:47.403 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:47.080+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:48.059 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:47.751+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:48.060 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:47.751+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:48.060 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:47.751+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:48.060 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:47.831+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:48.060 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:47.831+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:48.060 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:47.831+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:48.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:48.056+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:48.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:48.056+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:48.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:48.056+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:49.087 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:48.876+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:49.087 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:48.876+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:49.087 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:48.876+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:49.087 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:48.763+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:49.088 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:48.763+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:49.088 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:48.763+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:49.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:49.083+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:49.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:49.083+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:49.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:49.083+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:50.103 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:49.883+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:50.104 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:49.883+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:50.104 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:49.883+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:50.104 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:49.803+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:50.104 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:49.803+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:50.104 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:49.803+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:50.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:50.100+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:50.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:50.100+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:50.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:50.100+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:51.068 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:50.774+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:51.069 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:50.774+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:51.069 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:50.774+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:51.069 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:50.898+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:51.069 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:50.898+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:51.069 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:50.898+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:51.400 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:51.064+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:51.400 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:51.064+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:51.401 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:51.064+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:52.072 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:51.856+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:52.072 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:51.856+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:52.072 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:51.856+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:52.073 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:51.751+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:52.073 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:51.751+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:52.073 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:51.751+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:52.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:52.068+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:52.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:52.068+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:52.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:52.068+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:53.024 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:52.811+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:53.027 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:52.811+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:53.027 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:52.811+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:53.027 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:52.715+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:53.027 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:52.715+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:53.027 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:52.715+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:53.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:53.020+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:53.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:53.020+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:53.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:53.020+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:53.814 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:53.683+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:53.815 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:53.683+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:53.815 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:53.683+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:54.069 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:53.811+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:54.069 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:53.811+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:54.069 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:53.811+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:54.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:54.065+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:54.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:54.065+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:54.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:54.065+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:55.111 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:54.719+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:55.111 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:54.719+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:55.111 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:54.719+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:55.111 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:54.860+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:55.111 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:54.860+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:55.111 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:54.860+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:55.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:55.107+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:55.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:55.107+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:55.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:55.107+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:56.113 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:55.727+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:56.113 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:55.727+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:56.113 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:55.727+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:56.114 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:55.863+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:56.114 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:55.863+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:56.114 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:55.863+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:56.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:56.109+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:56.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:56.109+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:56.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:56.109+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:57.118 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:56.877+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:57.118 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:56.877+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:57.118 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:56.877+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:57.119 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:56.723+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:57.119 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:56.723+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:57.119 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:56.723+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:57.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:57.114+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:57.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:57.114+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:57.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:57.114+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:58.105 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:57.829+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:58.105 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:57.829+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:58.105 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:57.829+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:58.105 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:57.686+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:58.105 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:57.686+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:58.105 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:57.686+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:58.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:58.101+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:58.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:58.101+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:58.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:58.101+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:59.133 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:58.812+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:59.133 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:58.812+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:59.133 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:58.812+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:58:59.133 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:58.726+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:59.133 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:58.726+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:59.134 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:58.726+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:58:59.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:59.129+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:59.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:59.129+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:58:59.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:58:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:58:59.129+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:00.153 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:59.849+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:00.153 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:59.849+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:00.153 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:58:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:58:59.849+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:00.154 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:59.749+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:00.154 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:59.749+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:00.154 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:58:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:58:59.749+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:00.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:00.149+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:00.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:00.149+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:00.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:00.149+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:01.144 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:00.804+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:01.144 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:00.804+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:01.144 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:00.804+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:01.145 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:00.703+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:01.145 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:00.703+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:01.145 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:00.703+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:01.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:01.140+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:01.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:01.140+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:01.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:01.140+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:02.108 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:01.781+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:02.108 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:01.781+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:02.108 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:01.781+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:02.109 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:01.665+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:02.109 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:01.665+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:02.109 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:01.665+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:02.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:02.104+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:02.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:02.104+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:02.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:02.104+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:02.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:02.620+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:02.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:02.620+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:02.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:02.620+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:02.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:02.806+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:02.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:02.806+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:02.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:02.806+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:03.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:03.086+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:03.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:03.086+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:03.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:03.086+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:03.854 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:03.646+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:03.854 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:03.646+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:03.854 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:03.646+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:04.109 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:03.850+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:04.109 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:03.850+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:04.110 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:03.850+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:04.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:04.106+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:04.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:04.106+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:04.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:04.106+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:05.091 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:04.670+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:05.091 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:04.670+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:05.091 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:04.670+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:05.091 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:04.871+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:05.091 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:04.871+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:05.091 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:04.871+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:05.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:05.087+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:05.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:05.087+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:05.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:05.087+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:06.107 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:05.684+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:06.107 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:05.684+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:06.107 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:05.684+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:06.107 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:05.900+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:06.108 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:05.900+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:06.108 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:05.900+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:06.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:06.103+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:06.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:06.103+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:06.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:06.103+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:07.063 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:06.705+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:07.063 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:06.705+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:07.063 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:06.705+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:07.064 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:06.888+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:07.064 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:06.888+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:07.064 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:06.888+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:07.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:07.059+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:07.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:07.059+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:07.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:07.059+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:08.050 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:07.861+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:08.050 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:07.861+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:08.050 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:07.861+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:08.050 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:07.680+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:08.050 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:07.680+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:08.050 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:07.680+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:08.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:08.046+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:08.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:08.046+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:08.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:08.046+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:09.053 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:08.726+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:09.053 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:08.726+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:09.053 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:08.726+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:09.054 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:08.822+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:09.054 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:08.822+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:09.054 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:08.822+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:09.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:09.049+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:09.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:09.049+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:09.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:09.049+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:10.019 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:09.777+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:10.019 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:09.777+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:10.020 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:09.777+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:10.020 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:09.725+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:10.020 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:09.725+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:10.020 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:09.725+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:10.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:10.016+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:10.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:10.016+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:10.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:10.016+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:10.981 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:10.716+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:10.981 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:10.716+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:10.981 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:10.716+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:10.981 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:10.755+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:10.982 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:10.755+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:10.982 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:10.755+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:11.402 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:10.977+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:11.402 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:10.977+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:11.402 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:10.977+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:11.960 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:11.714+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:11.960 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:11.714+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:11.960 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:11.714+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:11.960 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:11.688+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:11.960 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:11.688+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:11.960 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:11.688+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:12.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:11.956+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:12.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:11.956+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:12.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:11.956+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:12.955 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:12.717+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:12.955 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:12.717+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:12.955 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:12.717+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:12.955 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:12.695+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:12.955 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:12.695+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:12.955 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:12.695+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:13.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:12.951+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:13.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:12.951+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:13.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:12.951+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:13.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:13.732+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:13.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:13.732+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:13.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:13.732+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:13.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:13.656+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:13.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:13.656+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:13.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:13.656+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:14.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:13.937+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:14.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:13.937+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:14.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:13.937+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:15.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:14.909+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:15.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:14.909+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:15.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:14.909+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:15.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:14.703+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:15.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:14.703+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:15.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:14.703+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:15.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:14.690+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:15.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:14.690+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:15.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:14.690+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:16.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:15.861+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:16.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:15.861+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:16.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:15.861+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:16.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:15.730+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:16.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:15.730+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:16.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:15.730+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:16.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:15.678+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:16.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:15.678+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:16.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:15.678+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:16.780 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:16.639+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:16.780 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:16.639+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:16.780 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:16.639+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:17.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:16.871+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:17.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:16.871+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:17.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:16.871+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:17.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:16.776+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:17.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:16.776+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:17.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:16.776+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:18.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:17.826+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:18.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:17.826+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:18.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:17.826+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:18.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:17.771+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:18.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:17.771+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:18.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:17.771+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:18.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:17.672+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:18.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:17.672+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:18.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:17.672+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:19.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:18.860+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:19.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:18.860+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:19.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:18.860+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:19.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:18.769+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:19.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:18.769+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:19.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:18.769+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:19.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:18.702+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:19.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:18.702+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:19.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:18.702+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:20.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:19.879+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:20.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:19.879+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:20.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:19.879+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:20.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:19.677+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:20.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:19.677+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:20.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:19.677+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:20.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:19.806+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:20.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:19.806+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:20.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:19.806+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:21.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:20.850+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:21.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:20.850+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:21.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:20.850+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:21.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:20.692+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:21.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:20.692+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:21.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:20.692+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:21.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:20.830+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:21.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:20.830+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:21.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:20.830+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:22.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:21.874+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:22.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:21.874+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:22.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:21.874+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:22.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:21.805+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:22.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:21.805+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:22.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:21.805+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:22.160 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:21.679+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:22.160 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:21.679+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:22.160 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:21.679+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:22.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:22.837+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:22.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:22.837+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:22.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:22.837+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:22.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:22.841+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:22.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:22.841+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:22.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:22.841+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:22.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:22.637+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:22.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:22.637+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:22.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:22.637+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:23.827 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:23.643+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:23.827 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:23.643+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:23.827 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:23.643+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:24.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:23.823+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:24.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:23.823+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:24.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:23.823+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:24.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:23.885+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:24.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:23.885+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:24.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:23.885+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:24.859 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:24.680+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:24.860 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:24.680+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:24.860 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:24.680+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:25.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:24.857+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:25.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:24.857+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:25.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:24.857+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:25.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:24.855+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:25.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:24.855+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:25.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:24.855+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:25.861 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:25.701+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:25.861 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:25.701+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:25.861 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:25.701+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:26.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:25.857+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:26.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:25.857+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:26.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:25.857+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:26.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:25.860+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:26.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:25.860+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:26.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:25.860+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:27.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:26.833+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:27.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:26.833+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:27.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:26.833+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:27.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:26.854+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:27.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:26.854+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:27.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:26.854+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:27.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:26.703+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:27.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:26.703+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:27.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:26.703+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:28.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:27.886+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:28.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:27.886+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:28.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:27.886+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:28.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:27.882+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:28.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:27.882+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:28.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:27.882+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:28.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:27.745+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:28.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:27.745+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:28.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:27.745+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:29.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:28.897+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:29.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:28.897+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:29.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:28.897+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:29.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:28.894+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:29.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:28.894+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:29.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:28.894+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:29.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:28.794+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:29.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:28.794+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:29.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:28.794+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:30.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:29.900+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:30.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:29.900+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:30.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:29.900+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:30.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:29.759+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:30.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:29.759+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:30.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:29.759+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:30.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:29.938+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:30.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:29.938+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:30.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:29.938+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:31.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:30.982+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:31.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:30.983+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:31.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:30.983+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:31.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:30.892+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:31.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:30.892+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:31.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:30.892+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:31.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:30.746+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:31.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:30.746+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:31.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:30.746+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:32.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:31.945+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:32.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:31.945+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:32.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:31.945+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:32.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:31.854+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:32.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:31.854+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:32.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:31.854+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:32.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:31.711+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:32.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:31.711+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:32.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:31.711+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:32.959 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:32.819+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:32.960 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:32.819+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:32.960 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:32.819+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:32.960 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:32.697+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:32.960 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:32.697+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:32.960 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:32.697+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:33.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:32.955+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:33.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:32.955+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:33.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:32.955+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:33.785 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:33.744+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:33.785 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:33.744+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:33.785 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:33.744+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:34.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:33.908+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:34.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:33.908+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:34.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:33.908+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:34.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:33.781+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:34.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:33.781+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:34.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:33.781+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:35.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:34.808+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:35.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:34.808+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:35.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:34.808+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:35.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:34.723+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:35.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:34.723+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:35.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:34.723+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:35.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:34.886+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:35.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:34.886+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:35.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:34.886+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:36.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:35.842+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:36.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:35.842+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:36.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:35.842+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:36.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:35.836+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:36.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:35.836+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:36.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:35.836+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:36.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:35.744+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:36.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:35.744+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:36.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:35.744+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:37.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:36.858+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:37.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:36.858+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:37.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:36.858+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:37.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:36.729+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:37.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:36.729+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:37.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:36.729+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:37.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:36.878+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:37.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:36.878+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:37.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:36.878+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:38.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:37.897+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:38.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:37.897+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:38.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:37.897+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:38.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:37.688+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:38.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:37.688+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:38.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:37.688+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:38.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:37.859+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:38.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:37.859+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:38.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:37.859+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:39.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:38.881+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:39.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:38.881+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:39.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:38.881+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:39.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:38.656+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:39.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:38.656+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:39.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:38.656+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:39.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:38.830+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:39.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:38.830+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:39.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:38.830+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:40.160 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:39.868+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:40.160 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:39.868+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:40.160 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:39.868+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:40.160 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:39.666+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:40.160 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:39.666+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:40.160 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:39.666+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:40.161 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:39.855+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:40.161 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:39.855+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:40.161 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:39.855+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:41.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:40.848+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:41.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:40.848+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:41.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:40.848+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:41.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:40.691+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:41.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:40.692+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:41.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:40.692+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:41.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:40.889+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:41.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:40.889+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:41.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:40.889+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:42.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:41.688+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:42.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:41.688+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:42.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:41.688+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:42.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:41.879+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:42.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:41.879+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:42.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:41.879+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:42.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:41.842+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:42.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:41.842+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:42.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:41.842+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:42.926 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:42.842+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:42.927 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:42.842+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:42.927 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:42.842+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:42.927 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:42.671+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:42.927 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:42.671+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:42.927 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:42.671+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:42.927 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:42.883+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:42.927 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:42.883+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:42.927 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:42.883+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:43.852 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:43.686+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:43.852 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:43.686+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:43.852 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:43.686+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:44.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:43.848+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:44.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:43.848+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:44.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:43.848+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:44.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:43.853+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:44.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:43.853+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:44.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:43.853+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:45.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:44.874+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:45.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:44.874+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:45.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:44.874+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:45.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:44.810+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:45.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:44.810+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:45.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:44.810+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:45.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:44.707+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:45.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:44.707+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:45.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:44.707+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:45.998 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:45.819+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:45.998 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:45.819+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:45.998 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:45.819+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:45.998 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:45.734+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:45.998 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:45.734+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:45.998 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:45.734+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:45.999 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:45.845+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:45.999 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:45.845+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:45.999 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:45.845+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:47.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:46.714+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:47.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:46.714+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:47.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:46.714+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:47.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:46.869+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:47.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:46.869+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:47.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:46.869+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:47.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:46.876+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:47.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:46.876+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:47.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:46.876+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:48.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:47.682+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:48.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:47.682+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:48.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:47.682+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:48.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:47.837+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:48.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:47.837+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:48.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:47.837+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:48.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:47.853+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:48.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:47.853+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:48.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:47.853+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:49.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:48.712+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:49.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:48.712+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:49.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:48.712+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:49.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:48.875+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:49.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:48.875+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:49.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:48.875+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:49.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:48.811+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:49.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:48.811+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:49.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:48.811+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:50.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:49.886+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:50.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:49.886+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:50.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:49.886+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:50.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:49.727+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:50.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:49.727+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:50.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:49.727+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:50.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:49.855+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:50.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:49.855+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:50.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:49.855+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:51.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:50.894+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:51.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:50.894+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:51.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:50.894+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:51.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:50.889+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:51.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:50.889+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:51.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:50.889+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:51.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:50.747+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:51.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:50.747+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:51.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:50.747+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:52.142 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:51.919+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:52.142 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:51.919+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:52.142 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:51.919+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:52.142 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:51.919+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:52.143 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:51.919+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:52.143 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:51.919+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:52.143 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:51.787+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:52.143 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:51.787+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:52.143 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:51.787+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:53.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:52.885+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:53.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:52.885+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:53.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:52.885+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:53.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:52.891+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:53.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:52.891+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:53.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:52.891+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:53.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:52.820+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:53.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:52.820+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:53.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:52.820+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:54.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:53.935+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:54.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:53.935+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:54.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:53.935+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:54.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:53.863+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:54.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:53.863+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:54.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:53.863+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:54.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:53.857+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:54.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:53.857+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:54.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:53.857+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:55.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:54.870+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:55.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:54.870+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:55.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:54.870+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:55.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:54.821+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:55.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:54.821+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:55.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:54.821+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:55.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:54.889+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:55.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:54.889+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:55.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:54.889+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:56.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:55.837+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:56.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:55.837+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:56.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:55.837+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:56.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:55.788+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:56.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:55.788+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:56.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:55.788+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:56.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:55.928+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:56.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:55.928+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:56.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:55.928+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:57.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:56.970+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:57.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:56.970+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:57.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:56.970+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:57.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:56.750+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:57.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:56.750+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:57.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:56.750+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:57.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:56.849+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:57.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:56.849+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:57.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:56.849+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:58.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:57.824+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:58.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:57.824+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:58.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:57.824+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:58.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:57.779+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:58.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:57.779+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:58.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:57.779+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:58.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:58.007+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:58.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:58.007+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:58.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:58.007+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:59.060 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:58.782+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:59.060 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:58.782+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:59.060 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:58.782+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T21:59:59.060 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:58.739+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:59.060 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:58.739+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:59.061 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:58.739+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T21:59:59.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:59.056+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:59.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:59.056+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T21:59:59.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 21:59:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T21:59:59.056+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:00.105 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:59.740+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:00.105 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:59.740+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:00.105 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 21:59:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T21:59:59.740+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:00.106 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:59.740+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:00.106 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:59.740+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:00.106 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 21:59:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T21:59:59.740+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:00.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:00.101+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:00.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:00.101+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:00.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:00.101+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:01.064 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:00.752+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:01.064 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:00.752+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:01.065 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:00.752+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:01.065 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:00.708+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:01.065 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:00.708+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:01.065 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:00.708+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:01.404 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:01.061+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:01.405 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:01.061+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:01.405 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:01.061+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:02.037 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:01.727+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:02.037 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:01.727+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:02.037 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:01.727+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:02.037 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:01.704+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:02.037 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:01.704+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:02.038 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:01.704+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:02.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:02.033+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:02.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:02.033+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:02.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:02.033+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:03.017 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:02.738+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:03.017 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:02.738+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:03.017 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:02.738+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:03.017 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:02.697+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:03.017 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:02.697+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:03.017 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:02.697+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:03.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:03.013+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:03.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:03.013+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:03.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:03.013+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:03.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:03.710+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:03.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:03.710+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:03.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:03.710+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:03.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:03.744+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:03.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:03.744+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:03.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:03.744+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:04.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:04.033+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:04.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:04.034+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:04.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:04.034+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:05.012 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:04.754+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:05.013 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:04.754+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:05.013 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:04.754+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:05.013 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:04.776+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:05.013 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:04.776+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:05.013 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:04.776+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:05.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:05.006+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:05.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:05.006+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:05.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:05.006+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:06.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:05.745+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:06.160 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:05.745+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:06.160 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:05.745+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:06.160 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:05.742+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:06.160 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:05.742+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:06.160 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:05.742+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:06.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:05.972+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:06.161 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:05.972+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:06.161 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:05.972+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:07.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:06.734+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:07.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:06.734+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:07.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:06.734+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:07.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:06.778+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:07.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:06.778+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:07.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:06.778+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:07.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:06.951+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:07.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:06.951+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:07.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:06.951+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:08.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:07.904+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:08.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:07.904+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:08.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:07.904+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:08.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:07.813+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:08.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:07.813+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:08.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:07.813+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:08.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:07.782+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:08.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:07.782+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:08.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:07.782+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:09.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:08.860+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:09.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:08.860+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:09.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:08.860+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:09.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:08.815+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:09.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:08.815+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:09.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:08.815+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:09.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:08.897+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:09.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:08.897+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:09.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:08.897+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:10.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:09.777+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:10.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:09.777+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:10.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:09.777+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:10.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:09.826+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:10.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:09.826+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:10.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:09.826+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:10.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:09.917+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:10.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:09.917+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:10.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:09.917+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:11.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:10.874+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:11.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:10.874+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:11.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:10.874+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:11.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:10.748+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:11.160 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:10.748+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:11.160 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:10.748+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:11.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:10.897+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:11.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:10.897+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:11.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:10.897+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:12.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:11.855+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:12.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:11.855+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:12.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:11.855+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:12.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:11.707+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:12.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:11.707+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:12.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:11.707+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:12.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:11.917+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:12.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:11.917+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:12.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:11.917+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:12.963 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:12.847+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:12.964 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:12.847+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:12.964 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:12.847+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:12.964 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:12.663+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:12.964 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:12.663+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:12.964 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:12.663+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:13.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:12.959+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:13.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:12.959+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:13.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:12.959+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:13.871 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:13.709+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:13.871 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:13.709+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:13.871 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:13.709+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:14.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:13.866+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:14.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:13.866+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:14.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:13.866+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:14.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:13.956+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:14.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:13.956+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:14.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:13.956+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:15.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:14.816+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:15.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:14.816+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:15.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:14.816+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:15.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:14.933+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:15.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:14.933+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:15.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:14.933+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:15.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:14.730+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:15.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:14.730+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:15.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:14.730+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:16.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:15.795+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:16.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:15.795+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:16.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:15.795+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:16.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:15.697+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:16.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:15.697+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:16.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:15.697+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:16.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:15.910+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:16.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:15.910+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:16.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:15.910+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:16.948 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:16.751+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:16.948 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:16.751+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:16.948 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:16.751+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:16.948 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:16.686+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:16.949 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:16.686+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:16.949 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:16.686+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:17.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:16.944+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:17.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:16.944+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:17.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:16.944+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:17.987 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:17.736+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:17.988 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:17.736+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:17.988 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:17.736+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:17.988 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:17.709+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:17.988 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:17.709+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:17.988 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:17.709+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:18.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:17.983+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:18.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:17.983+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:18.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:17.983+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:19.003 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:18.745+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:19.003 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:18.745+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:19.003 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:18.745+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:19.003 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:18.749+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:19.004 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:18.749+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:19.004 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:18.749+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:19.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:18.999+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:19.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:18.999+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:19.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:18.999+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:20.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:19.752+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:20.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:19.752+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:20.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:19.752+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:20.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:19.967+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:20.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:19.967+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:20.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:19.967+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:20.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:19.751+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:20.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:19.751+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:20.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:19.751+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:21.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:20.718+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:21.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:20.718+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:21.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:20.718+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:21.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:20.931+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:21.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:20.931+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:21.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:20.931+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:21.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:20.713+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:21.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:20.713+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:21.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:20.713+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:21.951 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:21.687+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:21.951 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:21.687+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:21.951 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:21.687+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:21.952 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:21.763+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:21.952 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:21.763+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:21.952 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:21.763+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:21.947+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:21.947+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:21.947+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:23.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:22.715+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:23.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:22.715+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:23.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:22.715+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:23.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:22.749+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:23.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:22.749+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:23.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:22.749+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:23.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:22.909+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:23.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:22.909+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:23.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:22.909+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:24.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:23.749+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:24.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:23.749+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:24.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:23.749+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:24.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:23.787+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:24.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:23.787+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:24.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:23.787+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:24.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:23.935+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:24.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:23.935+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:24.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:23.935+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:24.983 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:24.739+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:24.984 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:24.739+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:24.984 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:24.739+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:24.984 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:24.704+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:24.984 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:24.704+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:24.984 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:24.704+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:25.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:24.980+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:25.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:24.980+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:25.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:24.980+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:26.033 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:25.754+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:26.033 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:25.754+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:26.033 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:25.754+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:26.034 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:25.785+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:26.034 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:25.785+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:26.034 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:25.785+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:26.406 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:26.029+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:26.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:26.029+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:26.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:26.029+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:26.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:00:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:00:26.402+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 1 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:00:27.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:26.801+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:27.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:26.801+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:27.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:26.801+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:27.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:27.044+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:27.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:27.044+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:27.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:27.044+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:27.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:26.829+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:27.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:26.829+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:27.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:26.829+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:28.095 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:27.797+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:28.095 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:27.797+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:28.095 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:27.797+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:28.095 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:27.833+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:28.095 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:27.833+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:28.095 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:27.833+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:28.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:28.091+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:28.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:28.091+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:28.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:28.091+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:29.140 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:28.750+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:29.140 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:28.750+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:29.141 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:28.750+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:29.141 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:28.817+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:29.141 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:28.817+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:29.141 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:28.817+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:29.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:29.136+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:29.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:29.136+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:29.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:29.136+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:30.144 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:29.754+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:30.144 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:29.754+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:30.144 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:29.754+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:30.145 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:29.769+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:30.145 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:29.769+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:30.145 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:29.769+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:30.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:30.140+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:30.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:30.140+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:30.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:30.140+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:31.147 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:30.770+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:31.147 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:30.770+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:31.147 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:30.770+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:31.148 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:30.788+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:31.148 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:30.788+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:31.148 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:30.788+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:31.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:31.143+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:31.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:31.143+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:31.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:31.143+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:31.746 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:00:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:00:31.403+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 12 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:00:32.078 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:31.820+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:32.078 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:31.820+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:32.078 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:31.820+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:32.078 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:31.742+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:32.078 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:31.742+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:32.078 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:31.742+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:32.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:32.185+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:32.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:32.185+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:32.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:32.185+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:33.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:32.773+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:33.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:32.773+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:33.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:32.773+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:33.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:32.743+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:33.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:32.743+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:33.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:32.743+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:33.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:33.184+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:33.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:33.184+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:33.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:33.184+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:34.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:33.781+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:34.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:33.781+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:34.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:33.781+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:34.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:33.763+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:34.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:33.763+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:34.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:33.763+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:34.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:34.180+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:34.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:34.180+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:34.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:34.180+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:35.150 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:34.812+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:35.150 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:34.812+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:35.150 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:34.812+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:35.150 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:34.730+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:35.150 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:34.730+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:35.150 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:34.730+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:35.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:35.210+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:35.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:35.210+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:35.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:35.210+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:36.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:35.812+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:36.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:35.812+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:36.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:35.812+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:36.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:35.704+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:36.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:35.704+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:36.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:35.704+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:36.529 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:00:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:00:36.403+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 21 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:00:36.529 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:36.200+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:36.529 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:36.200+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:36.529 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:36.200+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:36.809 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:36.667+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:36.809 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:36.667+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:36.809 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:36.667+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:37.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:36.804+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:37.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:36.804+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:37.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:36.804+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:37.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:37.216+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:37.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:37.216+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:37.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:37.216+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:38.166 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:37.819+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:38.166 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:37.819+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:38.166 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:37.819+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:38.166 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:37.713+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:38.167 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:37.713+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:38.167 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:37.713+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:38.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:38.189+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:38.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:38.189+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:38.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:38.189+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:39.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:38.776+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:39.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:38.776+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:39.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:38.776+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:39.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:38.738+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:39.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:38.738+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:39.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:38.738+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:39.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:39.218+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:39.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:39.218+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:39.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:39.218+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:40.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:39.777+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:40.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:39.777+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:40.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:39.777+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:40.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:39.772+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:40.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:39.772+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:40.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:39.772+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:40.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:40.175+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:40.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:40.175+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:40.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:40.175+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:41.130 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:40.772+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:41.131 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:40.772+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:41.131 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:40.772+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:41.131 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:40.764+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:41.131 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:40.764+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:41.131 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:40.764+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:41.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:41.126+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:41.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:41.126+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:41.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:41.126+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:41.776 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:00:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:00:41.404+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 32 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:00:42.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:41.802+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:42.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:41.802+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:42.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:41.802+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:42.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:41.772+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:42.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:41.772+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:42.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:41.772+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:42.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:42.174+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:42.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:42.174+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:42.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:42.174+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:43.136 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:42.775+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:43.136 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:42.775+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:43.136 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:42.775+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:43.137 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:42.723+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:43.137 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:42.723+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:43.137 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:42.723+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:43.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:43.132+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:43.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:43.132+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:43.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:43.132+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:44.116 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:43.767+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:44.116 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:43.767+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:44.116 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:43.767+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:44.117 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:43.756+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:44.117 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:43.756+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:44.117 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:43.756+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:44.366 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:44.112+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:44.366 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:44.112+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:44.366 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:44.112+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:45.108 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:44.786+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:45.109 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:44.786+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:45.109 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:44.786+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:45.109 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:44.797+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:45.109 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:44.797+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:45.109 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:44.797+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:45.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:45.105+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:45.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:45.105+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:45.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:45.105+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:46.091 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:45.748+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:46.091 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:45.748+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:46.091 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:45.748+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:46.091 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:45.774+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:46.091 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:45.774+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:46.091 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:45.774+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:46.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:46.087+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:46.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:46.087+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:46.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:46.087+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:46.749 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:00:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:00:46.404+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 60 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:00:47.124 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:46.798+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:47.125 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:46.798+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:47.125 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:46.798+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:47.125 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:46.745+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:47.125 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:46.745+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:47.125 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:46.745+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:47.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:47.121+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:47.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:47.121+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:47.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:47.121+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:48.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:47.829+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:48.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:47.829+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:48.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:47.829+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:48.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:47.723+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:48.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:47.723+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:48.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:47.723+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:48.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:48.165+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:48.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:48.165+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:48.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:48.165+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:49.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:48.813+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:49.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:48.813+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:49.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:48.813+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:49.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:48.674+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:49.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:48.674+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:49.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:48.674+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:49.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:49.189+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:49.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:49.189+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:49.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:49.189+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:50.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:49.802+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:50.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:49.802+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:50.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:49.802+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:50.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:49.711+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:50.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:49.711+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:50.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:49.711+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:50.510 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:50.221+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:50.510 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:50.221+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:50.510 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:50.221+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:51.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:50.773+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:51.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:50.773+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:51.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:50.773+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:51.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:50.690+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:51.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:50.690+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:51.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:50.690+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:51.532 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:00:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:00:51.405+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 92 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:00:51.532 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:51.187+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:51.532 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:51.187+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:51.532 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:51.187+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:51.787 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:51.711+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:51.787 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:51.711+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:51.787 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:51.711+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:52.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:51.782+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:52.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:51.782+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:52.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:51.782+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:52.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:52.163+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:52.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:52.163+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:52.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:52.163+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:53.119 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:52.802+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:53.120 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:52.802+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:53.120 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:52.802+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:53.120 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:52.745+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:53.120 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:52.745+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:53.120 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:52.745+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:53.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:53.115+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:53.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:53.115+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:53.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:53.115+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:54.156 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:53.850+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:54.156 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:53.850+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:54.156 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:53.850+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:54.156 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:53.791+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:54.156 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:53.791+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:54.156 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:53.791+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:54.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:54.152+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:54.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:54.152+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:54.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:54.152+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:55.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:54.800+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:55.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:54.800+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:55.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:54.800+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:55.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:54.828+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:55.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:54.828+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:55.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:54.828+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:55.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:55.200+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:55.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:55.200+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:55.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:55.200+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:56.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:55.835+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:56.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:55.835+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:56.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:55.835+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:56.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:55.814+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:56.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:55.814+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:56.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:55.814+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:56.431 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:00:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:00:56.405+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 119 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:00:56.432 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:56.168+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:56.432 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:56.168+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:56.432 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:56.168+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:57.123 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:56.768+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:57.123 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:56.768+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:57.123 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:56.768+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:57.123 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:56.815+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:57.124 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:56.815+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:57.124 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:56.815+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:57.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:57.119+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:57.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:57.119+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:57.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:57.119+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:58.123 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:57.807+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:58.123 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:57.807+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:58.123 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:57.807+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:58.123 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:57.809+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:58.123 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:57.809+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:58.123 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:57.809+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:58.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:58.119+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:58.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:58.119+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:58.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:58.119+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:59.139 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:58.852+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:59.139 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:58.852+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:59.139 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:58.852+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:00:59.139 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:58.774+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:59.139 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:58.774+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:59.139 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:58.774+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:00:59.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:59.135+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:59.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:59.135+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:00:59.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:00:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:00:59.135+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:00.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:59.817+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:00.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:59.817+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:00.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:00:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:00:59.817+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:00.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:59.747+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:00.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:59.747+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:00.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:00:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:00:59.747+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:00.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:00.155+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:00.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:00.155+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:00.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:00.155+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:01.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:00.846+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:01.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:00.846+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:01.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:00.846+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:01.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:00.776+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:01.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:00.776+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:01.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:00.776+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:01.532 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:01:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:01:01.405+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 148 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:01:01.533 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:01.185+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:01.533 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:01.185+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:01.533 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:01.185+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:01.831 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:01.734+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:01.831 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:01.734+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:01.832 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:01.734+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:02.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:01.835+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:02.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:01.835+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:02.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:01.835+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:02.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:02.158+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:02.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:02.158+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:02.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:02.158+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:03.145 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:02.791+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:03.145 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:02.791+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:03.145 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:02.791+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:03.145 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:02.719+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:03.145 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:02.719+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:03.145 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:02.719+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:03.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:03.141+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:03.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:03.141+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:03.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:03.141+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:03.805 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:03.670+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:03.805 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:03.670+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:03.805 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:03.671+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:04.154 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:03.801+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:04.154 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:03.801+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:04.154 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:03.801+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:04.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:04.150+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:04.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:04.150+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:04.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:04.150+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:05.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:04.806+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:05.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:04.806+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:05.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:04.806+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:05.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:04.696+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:05.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:04.696+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:05.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:04.696+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:05.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:05.158+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:05.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:05.158+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:05.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:05.158+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:06.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:05.837+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:06.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:05.837+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:06.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:05.837+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:06.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:05.719+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:06.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:05.719+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:06.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:05.719+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:06.420 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:01:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:01:06.406+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 174 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:01:06.420 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:06.161+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:06.420 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:06.161+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:06.420 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:06.161+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:07.132 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:06.873+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:07.133 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:06.873+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:07.133 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:06.873+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:07.133 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:06.740+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:07.133 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:06.740+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:07.133 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:06.740+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:07.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:07.129+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:07.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:07.129+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:07.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:07.129+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:08.129 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:07.916+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:08.129 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:07.916+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:08.129 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:07.916+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:08.130 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:07.722+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:08.130 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:07.722+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:08.130 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:07.722+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:08.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:08.125+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:08.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:08.125+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:08.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:08.125+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:09.114 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:08.910+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:09.114 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:08.910+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:09.114 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:08.910+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:09.115 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:08.679+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:09.115 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:08.679+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:09.115 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:08.679+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:09.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:09.110+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:09.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:09.110+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:09.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:09.110+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:09.905 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:09.633+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:09.905 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:09.633+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:09.905 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:09.633+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:10.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:09.901+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:10.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:09.901+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:10.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:09.901+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:10.641 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:10.159+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:10.641 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:10.159+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:10.641 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:10.159+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:10.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:10.637+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:10.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:10.637+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:10.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:10.637+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:11.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:10.904+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:11.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:10.904+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:11.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:10.904+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:11.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:11.123+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:11.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:11.123+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:11.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:11.123+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:11.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:01:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:01:11.406+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 207 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:01:11.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:11.600+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:11.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:11.600+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:11.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:11.600+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:12.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:11.916+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:12.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:11.916+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:12.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:11.916+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:12.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:12.123+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:12.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:12.123+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:12.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:12.123+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:12.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:12.558+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:12.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:12.558+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:12.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:12.558+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:13.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:12.909+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:13.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:12.909+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:13.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:12.909+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:13.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:13.131+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:13.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:13.131+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:13.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:13.131+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:13.897 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:13.604+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:13.897 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:13.604+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:13.897 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:13.604+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:14.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:13.891+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:14.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:13.891+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:14.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:13.892+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:14.565 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:14.162+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:14.565 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:14.162+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:14.565 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:14.162+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:14.899 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:14.561+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:14.899 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:14.561+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:14.899 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:14.561+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:15.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:14.895+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:15.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:14.895+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:15.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:14.895+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:15.585 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:15.206+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:15.585 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:15.206+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:15.585 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:15.206+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:15.877 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:15.580+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:15.878 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:15.580+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:15.878 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:15.580+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:16.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:15.873+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:16.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:15.873+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:16.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:15.873+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:16.534 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:01:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:01:16.407+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 234 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:01:16.535 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:16.193+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:16.535 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:16.193+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:16.535 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:16.193+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:16.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:16.532+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:16.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:16.532+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:16.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:16.532+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:17.241 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:16.918+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:17.241 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:16.918+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:17.241 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:16.918+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:17.510 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:17.237+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:17.510 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:17.237+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:17.510 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:17.237+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:17.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:17.506+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:17.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:17.506+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:17.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:17.506+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:18.270 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:17.963+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:18.270 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:17.963+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:18.270 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:17.963+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:18.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:18.266+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:18.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:18.266+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:18.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:18.266+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:18.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:18.478+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:18.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:18.478+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:18.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:18.478+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:19.274 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:18.993+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:19.274 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:18.993+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:19.275 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:18.993+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:19.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:19.271+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:19.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:19.271+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:19.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:19.271+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:19.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:19.463+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:19.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:19.463+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:19.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:19.463+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:20.278 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:20.016+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:20.278 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:20.016+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:20.278 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:20.016+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:20.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:20.418+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:20.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:20.418+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:20.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:20.418+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:20.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:20.274+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:20.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:20.274+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:20.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:20.274+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:21.383 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:21.060+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:21.383 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:21.060+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:21.383 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:21.060+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:21.383 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:21.287+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:21.384 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:21.287+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:21.384 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:21.287+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:21.635 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:01:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:01:21.407+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 262 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:01:21.635 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:21.379+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:21.635 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:21.379+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:21.635 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:21.379+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:22.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:22.330+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:22.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:22.330+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:22.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:22.330+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:22.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:22.088+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:22.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:22.088+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:22.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:22.088+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:22.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:22.291+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:22.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:22.291+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:22.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:22.291+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:23.362 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:23.243+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:23.362 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:23.243+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:23.363 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:23.243+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:23.363 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:23.090+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:23.363 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:23.090+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:23.363 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:23.090+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:23.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:23.358+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:23.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:23.358+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:23.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:23.359+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:24.382 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:24.093+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:24.382 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:24.093+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:24.382 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:24.093+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:24.382 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:24.221+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:24.382 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:24.221+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:24.382 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:24.221+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:24.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:24.378+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:24.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:24.378+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:24.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:24.378+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:25.339 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:25.047+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:25.339 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:25.047+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:25.339 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:25.047+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:25.339 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:25.266+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:25.339 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:25.266+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:25.339 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:25.266+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:25.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:25.335+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:25.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:25.335+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:25.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:25.335+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:26.306 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:26.050+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:26.306 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:26.050+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:26.306 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:26.050+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:26.306 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:26.265+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:26.306 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:26.265+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:26.306 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:26.265+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:26.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:01:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:01:26.407+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 289 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:01:26.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:26.302+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:26.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:26.302+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:26.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:26.302+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:27.343 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:27.052+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:27.343 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:27.052+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:27.344 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:27.052+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:27.344 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:27.248+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:27.344 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:27.248+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:27.344 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:27.248+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:27.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:27.339+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:27.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:27.340+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:27.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:27.340+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:28.363 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:28.040+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:28.363 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:28.040+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:28.363 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:28.040+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:28.363 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:28.041+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:277 2.13 2:ceffbd45:::200.0000000a:head [write 751333~1450 in=1450b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:28.364 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:28.220+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:28.364 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:28.220+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:28.364 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:28.220+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:28.660 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:28.359+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:28.660 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:28.359+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:28.660 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:28.359+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:29.373 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:28.993+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:29.373 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:28.993+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:29.373 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:28.993+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:29.373 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:28.993+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:277 2.13 2:ceffbd45:::200.0000000a:head [write 751333~1450 in=1450b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:29.373 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:29.200+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:29.373 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:29.200+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:29.373 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:29.200+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:29.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:29.369+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:29.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:29.369+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:29.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:29.369+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:29.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:29.370+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:30.332 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:30.005+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:30.332 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:30.005+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:30.332 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:30.005+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:30.332 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:30.007+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:30.333 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:30.171+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:30.333 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:30.171+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:30.333 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:30.171+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:30.333 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:30.173+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:01:30.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:30.328+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:30.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:30.328+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:30.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:30.328+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:30.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:30.329+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:31.283 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:31.003+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:31.283 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:31.003+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:31.284 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:31.003+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:31.284 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:31.004+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:31.284 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:31.216+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:31.284 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:31.216+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:31.284 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:31.216+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:31.284 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:31.217+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:01:31.536 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:01:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:01:31.408+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 318 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:01:31.536 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:31.279+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:31.536 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:31.279+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:31.536 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:31.279+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:31.536 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:31.280+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:32.259 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:31.987+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:32.259 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:31.987+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:32.260 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:31.987+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:32.260 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:31.988+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:32.260 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:32.193+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:32.260 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:32.193+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:32.260 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:32.193+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:32.260 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:32.194+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:01:32.260 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:32.255+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:32.260 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:32.255+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:32.261 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:32.255+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:32.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:32.256+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:33.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:33.028+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:33.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:33.028+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:33.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:33.028+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:33.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:33.029+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:33.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:33.193+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:33.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:33.193+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:33.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:33.193+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:33.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:33.194+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:01:33.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:33.248+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:33.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:33.248+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:33.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:33.248+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:33.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:33.248+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:34.287 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:34.027+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:34.287 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:34.027+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:34.287 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:34.027+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:34.287 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:34.028+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:34.287 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:34.150+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:34.288 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:34.150+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:34.288 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:34.150+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:34.288 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:34.152+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:01:34.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:34.283+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:34.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:34.283+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:34.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:34.283+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:34.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:34.284+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:35.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:35.043+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:35.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:35.043+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:35.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:35.043+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:35.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:35.044+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:35.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:35.162+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:35.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:35.162+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:35.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:35.162+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:35.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:35.163+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:01:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:35.262+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:35.262+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:35.262+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:35.263+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:36.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:36.047+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:36.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:36.047+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:36.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:36.047+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:36.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:36.048+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:36.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:36.161+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:36.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:36.161+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:36.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:36.161+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:36.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:36.162+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:01:36.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:36.247+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:36.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:36.247+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:36.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:36.247+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:36.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:36.248+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:36.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:01:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:01:36.408+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 346 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:01:37.277 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:37.073+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:37.277 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:37.073+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:37.277 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:37.073+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:37.277 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:37.074+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:37.277 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:37.127+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:37.277 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:37.127+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:37.277 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:37.127+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:37.277 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:37.128+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:01:37.278 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:37.273+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:37.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:37.273+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:37.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:37.273+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:37.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:37.274+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:38.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:38.036+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:38.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:38.036+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:38.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:38.036+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:38.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:38.037+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:38.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:38.264+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:38.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:38.264+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:38.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:38.264+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:38.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:38.265+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:38.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:38.156+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:38.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:38.156+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:38.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:38.156+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:38.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:38.157+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:01:39.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:39.034+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:39.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:39.034+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:39.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:39.034+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:39.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:39.036+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:39.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:39.278+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:39.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:39.278+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:39.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:39.278+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:39.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:39.279+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:39.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:39.142+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:39.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:39.142+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:39.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:39.142+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:39.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:39.143+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:01:40.302 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:40.014+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:40.302 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:40.014+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:40.302 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:40.014+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:40.302 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:40.014+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:40.302 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:40.105+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:40.303 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:40.105+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:40.303 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:40.105+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:40.303 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:40.106+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:01:40.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:40.298+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:40.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:40.298+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:40.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:40.298+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:40.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:40.299+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:41.315 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:41.039+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:41.315 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:41.039+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:41.315 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:41.039+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:41.315 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:41.040+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:41.316 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:41.147+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:41.316 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:41.147+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:41.316 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:41.147+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:41.316 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:41.148+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:01:41.636 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:01:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:01:41.409+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 377 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:01:41.637 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:41.310+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:41.637 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:41.310+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:41.637 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:41.310+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:41.637 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:41.311+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:42.314 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:41.994+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:42.314 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:41.994+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:42.314 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:41.994+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:42.314 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:41.995+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:42.315 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:42.150+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:42.315 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:42.150+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:42.315 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:42.150+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:42.315 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:42.151+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:01:42.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:42.309+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:42.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:42.309+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:42.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:42.309+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:42.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:42.310+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:43.272 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:43.158+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:43.272 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:43.158+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:43.272 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:43.158+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:43.272 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:43.159+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:01:43.273 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:42.981+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:43.273 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:42.981+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:43.273 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:42.981+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:43.273 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:42.982+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:43.523 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:43.267+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:43.524 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:43.267+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:43.524 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:43.267+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:43.524 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:43.268+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:44.322 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:44.001+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:44.322 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:44.001+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:44.322 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:44.001+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:44.322 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:44.002+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:44.322 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:44.132+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:44.322 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:44.132+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:44.323 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:44.132+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:44.323 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:44.133+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:01:44.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:44.317+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:44.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:44.317+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:44.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:44.317+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:44.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:44.318+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:45.299 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:44.994+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:45.299 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:44.994+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:45.299 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:44.994+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:45.299 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:44.994+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:45.300 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:45.112+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:45.300 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:45.112+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:45.300 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:45.112+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:45.300 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:45.113+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:01:45.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:45.294+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:45.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:45.294+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:45.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:45.294+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:45.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:45.295+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:46.106 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:46.002+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:46.106 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:46.002+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:46.106 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:46.002+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:46.106 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:46.003+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:46.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:46.262+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:46.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:46.262+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:46.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:46.262+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:46.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:46.263+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:46.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:46.101+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:46.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:46.101+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:46.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:46.101+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:46.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:46.102+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:01:46.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:01:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:01:46.408+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 408 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:01:47.237 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:47.100+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:47.238 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:47.100+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:47.238 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:47.100+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:47.238 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:47.101+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:01:47.238 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:46.971+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:47.238 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:46.971+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:47.238 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:46.971+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:47.238 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:46.972+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:47.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:47.232+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:47.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:47.232+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:47.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:47.233+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:47.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:47.234+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:48.203 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:47.944+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:48.203 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:47.944+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:48.203 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:47.944+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:48.203 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:47.945+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:48.203 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:48.071+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:48.203 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:48.071+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:48.203 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:48.072+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:48.204 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:48.073+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:01:48.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:48.198+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:48.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:48.198+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:48.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:48.198+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:48.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:48.199+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:49.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:48.917+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:49.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:48.917+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:49.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:48.917+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:49.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:48.918+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:49.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:49.036+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:49.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:49.036+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:49.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:49.036+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:49.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:49.037+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:01:49.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:49.198+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:49.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:49.198+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:49.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:49.198+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:49.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:49.199+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:50.216 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:49.935+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:50.217 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:49.935+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:50.217 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:49.935+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:50.217 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:49.936+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:50.217 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:50.071+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:50.217 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:50.071+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:50.218 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:50.071+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:50.218 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:50.072+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:01:50.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:50.211+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:50.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:50.211+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:50.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:50.211+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:50.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:50.212+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:51.198 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:50.904+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:51.198 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:50.904+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:51.198 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:50.904+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:51.198 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:50.905+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:51.199 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:51.103+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:51.199 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:51.103+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:51.199 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:51.103+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:51.199 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:51.104+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:01:51.538 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:01:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:01:51.410+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 441 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:01:51.538 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:51.194+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:51.538 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:51.194+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:51.538 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:51.194+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:51.538 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:51.195+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:52.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:52.072+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:52.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:52.072+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:52.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:52.072+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:52.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:52.072+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:01:52.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:51.863+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:52.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:51.863+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:52.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:51.863+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:52.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:51.864+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:52.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:52.228+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:52.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:52.228+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:52.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:52.228+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:52.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:52.229+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:53.121 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:52.849+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:53.121 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:52.849+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:53.121 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:52.849+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:53.121 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:52.850+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:53.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:53.238+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:53.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:53.238+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:53.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:53.238+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:53.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:53.239+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:53.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:53.117+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:53.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:53.117+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:53.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:53.117+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:53.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:53.118+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:01:54.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:54.140+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:54.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:54.140+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:54.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:54.140+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:54.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:54.141+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:01:54.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:53.894+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:54.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:53.894+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:54.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:53.894+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:54.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:53.895+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:54.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:54.212+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:54.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:54.212+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:54.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:54.212+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:54.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:54.214+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:55.185 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:55.123+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:55.185 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:55.123+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:55.185 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:55.123+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:55.185 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:55.124+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:01:55.185 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:54.916+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:55.186 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:54.916+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:55.186 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:54.916+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:55.186 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:54.917+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:55.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:55.181+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:55.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:55.181+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:55.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:55.181+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:55.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:55.182+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:56.218 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:55.926+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:56.218 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:55.926+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:56.218 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:55.926+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:56.218 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:55.926+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:56.218 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:56.079+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:56.218 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:56.079+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:56.218 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:56.079+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:56.218 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:56.080+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:01:56.538 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:01:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:01:56.410+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 469 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:01:56.538 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:56.213+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:56.538 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:56.214+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:56.539 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:56.214+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:56.539 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:56.215+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:57.234 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:57.125+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:57.234 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:57.125+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:57.234 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:57.125+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:57.234 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:57.125+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:01:57.234 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:56.937+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:57.234 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:56.937+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:57.234 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:56.937+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:57.235 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:56.938+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:57.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:57.230+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:57.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:57.230+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:57.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:57.230+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:57.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:57.231+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:58.228 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:58.082+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:58.229 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:58.082+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:58.229 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:58.082+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:58.229 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:58.082+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:01:58.229 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:57.908+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:58.229 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:57.908+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:58.229 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:57.908+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:58.229 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:57.908+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:58.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:58.224+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:58.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:58.224+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:58.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:58.224+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:58.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:58.225+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:59.231 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:58.951+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:59.231 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:58.951+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:59.231 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:58.951+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:01:59.231 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:58.952+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:01:59.231 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:59.107+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:59.231 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:59.107+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:59.231 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:59.107+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:01:59.231 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:01:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:01:59.107+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:01:59.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:59.227+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:59.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:59.227+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:59.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:59.227+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:01:59.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:01:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:01:59.228+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:00.254 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:59.960+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:00.254 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:59.960+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:00.254 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:59.960+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:00.254 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:01:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:01:59.961+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:00.255 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:00.080+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:00.255 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:00.080+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:00.255 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:00.080+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:00.255 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:00.081+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:00.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:00.250+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:00.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:00.250+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:00.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:00.250+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:00.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:00.251+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:01.258 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:00.978+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:01.258 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:00.978+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:01.258 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:00.978+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:01.259 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:00.978+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:01.259 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:01.104+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:01.259 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:01.104+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:01.259 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:01.104+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:01.259 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:01.104+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:01.538 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:02:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:02:01.411+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 497 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:02:01.538 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:01.254+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:01.538 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:01.254+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:01.538 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:01.254+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:01.539 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:01.255+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:02.307 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:02.010+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:02.308 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:02.010+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:02.308 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:02.010+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:02.308 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:02.011+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:02.308 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:02.113+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:02.308 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:02.113+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:02.308 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:02.113+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:02.308 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:02.114+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:02.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:02.303+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:02.660 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:02.303+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:02.660 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:02.303+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:02.660 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:02.304+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:03.316 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:02.992+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:03.316 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:02.992+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:03.316 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:02.992+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:03.316 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:02.992+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:03.316 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:03.082+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:03.316 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:03.082+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:03.317 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:03.082+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:03.317 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:03.083+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:03.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:03.312+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:03.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:03.312+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:03.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:03.312+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:03.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:03.313+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:04.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:04.038+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:04.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:04.038+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:04.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:04.038+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:04.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:04.039+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:04.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:04.031+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:04.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:04.031+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:04.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:04.031+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:04.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:04.032+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:04.273+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:04.273+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:04.273+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:04.274+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:05.268 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:04.983+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:05.268 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:04.983+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:05.268 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:04.983+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:05.268 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:04.984+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:05.268 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:05.031+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:05.269 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:05.031+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:05.269 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:05.031+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:05.269 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:05.032+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:05.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:05.264+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:05.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:05.264+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:05.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:05.264+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:05.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:05.265+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:06.309 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:06.001+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:06.310 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:06.002+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:06.310 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:06.002+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:06.310 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:06.003+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:06.310 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:05.964+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:06.310 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:05.964+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:06.311 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:05.964+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:06.311 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:05.965+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:06.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:02:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:02:06.411+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 525 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:02:06.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:06.305+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:06.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:06.305+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:06.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:06.305+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:06.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:06.306+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:07.297 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:06.987+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:07.297 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:06.987+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:07.297 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:06.987+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:07.298 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:06.988+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:07.298 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:06.995+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:07.298 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:06.995+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:07.298 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:06.995+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:07.298 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:06.995+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:07.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:07.293+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:07.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:07.293+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:07.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:07.293+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:07.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:07.294+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:08.284 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:08.011+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:08.284 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:08.011+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:08.284 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:08.011+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:08.284 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:08.012+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:08.284 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:07.955+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:08.284 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:07.955+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:08.284 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:07.955+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:08.285 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:07.956+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:08.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:08.280+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:08.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:08.280+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:08.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:08.280+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:08.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:08.281+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:09.316 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:09.048+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:09.316 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:09.048+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:09.316 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:09.048+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:09.316 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:09.049+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:09.317 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:08.935+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:09.317 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:08.935+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:09.317 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:08.935+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:09.317 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:08.936+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:09.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:09.312+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:09.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:09.312+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:09.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:09.312+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:09.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:09.313+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:10.340 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:10.036+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:10.340 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:10.036+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:10.340 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:10.036+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:10.340 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:10.037+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:10.341 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:09.955+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:10.341 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:09.955+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:10.341 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:09.955+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:10.341 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:09.956+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:10.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:10.336+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:10.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:10.336+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:10.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:10.336+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:10.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:10.337+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:11.344 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:10.989+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:11.344 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:10.989+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:11.344 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:10.989+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:11.344 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:10.989+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:11.345 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:10.951+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:11.345 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:10.951+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:11.345 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:10.951+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:11.345 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:10.952+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:11.638 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:02:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:02:11.412+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 555 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:02:11.638 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:11.340+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:11.639 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:11.340+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:11.639 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:11.340+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:11.639 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:11.341+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:12.302 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:12.031+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:12.303 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:12.031+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:12.303 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:12.031+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:12.303 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:12.032+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:12.303 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:11.919+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:12.303 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:11.919+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:12.303 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:11.919+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:12.303 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:11.919+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:12.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:12.299+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:12.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:12.299+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:12.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:12.299+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:12.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:12.300+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:13.296 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:13.072+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:13.296 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:13.072+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:13.296 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:13.072+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:13.296 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:13.073+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:13.297 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:12.964+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:13.297 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:12.964+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:13.297 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:12.964+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:13.297 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:12.965+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:13.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:13.292+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:13.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:13.292+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:13.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:13.292+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:13.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:13.293+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:14.302 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:14.026+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:14.302 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:14.026+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:14.303 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:14.026+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:14.303 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:14.026+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:14.303 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:13.984+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:14.303 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:13.984+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:14.303 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:13.984+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:14.303 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:13.985+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:14.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:14.298+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:14.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:14.298+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:14.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:14.298+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:14.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:14.299+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:15.327 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:15.038+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:15.327 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:15.038+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:15.327 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:15.038+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:15.327 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:15.039+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:15.327 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:14.995+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:15.328 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:14.995+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:15.328 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:14.995+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:15.328 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:14.996+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:15.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:15.323+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:15.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:15.323+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:15.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:15.323+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:15.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:15.324+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:16.344 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:16.079+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:16.344 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:16.079+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:16.344 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:16.079+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:16.344 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:16.080+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:16.345 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:16.035+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:16.345 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:16.035+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:16.345 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:16.035+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:16.345 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:16.037+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:16.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:02:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:02:16.412+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 583 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:02:16.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:16.340+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:16.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:16.340+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:16.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:16.340+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:16.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:16.341+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:17.319 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:17.063+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:17.319 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:17.063+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:17.319 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:17.063+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:17.319 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:17.064+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:17.319 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:17.059+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:17.319 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:17.059+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:17.320 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:17.059+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:17.320 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:17.060+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:17.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:17.315+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:17.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:17.315+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:17.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:17.315+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:17.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:17.316+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:18.307 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:18.046+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:18.307 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:18.046+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:18.307 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:18.046+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:18.307 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:18.047+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:18.307 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:18.033+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:18.307 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:18.033+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:18.308 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:18.033+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:18.308 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:18.034+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:18.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:18.302+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:18.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:18.302+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:18.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:18.302+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:18.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:18.303+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:19.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:19.047+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:19.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:19.047+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:19.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:19.048+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:19.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:19.049+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:19.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:19.078+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:19.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:19.078+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:19.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:19.078+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:19.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:19.079+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:19.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:19.273+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:19.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:19.273+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:19.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:19.273+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:19.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:19.273+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:20.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:20.038+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:20.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:20.038+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:20.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:20.038+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:20.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:20.039+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:20.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:20.129+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:20.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:20.129+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:20.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:20.129+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:20.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:20.130+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:20.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:20.267+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:20.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:20.267+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:20.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:20.267+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:20.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:20.267+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:21.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:21.119+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:21.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:21.119+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:21.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:21.119+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:21.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:21.120+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:21.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:21.056+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:21.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:21.056+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:21.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:21.056+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:21.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:21.057+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:21.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:21.226+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:21.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:21.226+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:21.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:21.226+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:21.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:21.227+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:21.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:02:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:02:21.413+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 611 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:02:22.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:22.193+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:22.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:22.193+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:22.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:22.193+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:22.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:22.193+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:22.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:22.082+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:22.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:22.082+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:22.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:22.082+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:22.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:22.083+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:22.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:22.102+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:22.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:22.102+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:22.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:22.102+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:22.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:22.103+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:23.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:23.102+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:23.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:23.102+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:23.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:23.102+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:23.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:23.103+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:23.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:23.225+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:23.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:23.225+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:23.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:23.225+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:23.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:23.226+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:23.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:23.075+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:23.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:23.075+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:23.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:23.075+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:23.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:23.076+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:24.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:24.079+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:24.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:24.079+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:24.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:24.079+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:24.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:24.080+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:24.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:24.181+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:24.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:24.181+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:24.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:24.181+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:24.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:24.181+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:24.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:24.047+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:24.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:24.047+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:24.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:24.047+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:24.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:24.048+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:25.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:25.049+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:25.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:25.049+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:25.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:25.049+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:25.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:25.049+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:25.411 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:25.174+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:25.411 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:25.174+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:25.411 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:25.174+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:25.411 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:25.175+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:25.412 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:25.021+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:25.412 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:25.021+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:25.412 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:25.021+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:25.412 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:25.022+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:26.019 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:25.976+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:26.019 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:25.976+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:26.020 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:25.976+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:26.020 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:25.977+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:26.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:26.015+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:26.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:26.015+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:26.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:26.015+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:26.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:26.016+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:26.201+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:26.201+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:26.201+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:26.202+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:26.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:02:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:02:26.413+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 638 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:02:27.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:26.983+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:27.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:26.983+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:27.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:26.983+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:27.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:26.984+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:27.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:27.215+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:27.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:27.215+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:27.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:27.215+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:27.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:27.216+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:27.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:26.968+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:27.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:26.968+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:27.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:26.968+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:27.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:26.968+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:28.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:27.990+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:28.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:27.990+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:28.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:27.990+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:28.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:27.991+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:28.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:28.197+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:28.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:28.197+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:28.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:28.197+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:28.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:28.198+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:28.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:28.000+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:28.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:28.000+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:28.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:28.000+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:28.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:28.001+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:29.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:29.029+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:29.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:29.029+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:29.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:29.029+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:29.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:29.030+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:29.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:29.242+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:29.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:29.242+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:29.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:29.242+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:29.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:29.243+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:29.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:29.010+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:29.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:29.010+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:29.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:29.010+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:29.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:29.011+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:30.286 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:30.060+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:30.287 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:30.060+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:30.287 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:30.060+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:30.287 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:30.061+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:30.287 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:29.993+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:30.287 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:29.993+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:30.287 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:29.993+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:30.287 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:29.994+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:30.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:30.282+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:30.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:30.282+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:30.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:30.282+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:30.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:30.283+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:31.311 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:31.044+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:31.311 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:31.044+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:31.311 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:31.044+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:31.311 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:31.045+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:31.312 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:31.014+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:31.312 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:31.014+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:31.312 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:31.014+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:31.312 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:31.015+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:31.639 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:02:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:02:31.414+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 667 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:02:31.640 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:31.307+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:31.640 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:31.307+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:31.640 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:31.307+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:31.640 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:31.308+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:32.316 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:32.088+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:32.316 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:32.088+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:32.316 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:32.088+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:32.316 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:32.089+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:32.317 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:32.051+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:32.317 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:32.052+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:32.317 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:32.052+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:32.317 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:32.053+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:32.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:32.312+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:32.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:32.312+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:32.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:32.312+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:32.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:32.313+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:33.307 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:33.051+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:33.307 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:33.051+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:33.308 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:33.051+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:33.308 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:33.052+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:33.308 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:33.060+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:33.308 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:33.060+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:33.308 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:33.060+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:33.308 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:33.061+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:33.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:33.303+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:33.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:33.303+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:33.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:33.303+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:33.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:33.304+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:34.329 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:34.070+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:34.329 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:34.070+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:34.329 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:34.070+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:34.329 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:34.071+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:34.329 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:34.057+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:34.329 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:34.057+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:34.329 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:34.057+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:34.329 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:34.057+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:34.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:34.325+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:34.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:34.325+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:34.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:34.325+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:34.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:34.326+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:35.333 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:35.113+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:35.333 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:35.113+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:35.333 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:35.113+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:35.333 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:35.114+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:35.333 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:35.104+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:35.333 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:35.104+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:35.333 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:35.104+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:35.334 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:35.105+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:35.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:35.329+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:35.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:35.329+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:35.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:35.329+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:35.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:35.330+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:36.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:36.153+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:36.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:36.153+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:36.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:36.153+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:36.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:36.154+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:36.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:36.102+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:36.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:36.102+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:36.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:36.102+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:36.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:36.104+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:36.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:36.350+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:36.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:36.350+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:36.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:36.350+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:36.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:36.351+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:36.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:02:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:02:36.414+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 695 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:02:37.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:37.180+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:37.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:37.180+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:37.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:37.180+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:37.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:37.180+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:37.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:37.330+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:37.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:37.331+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:37.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:37.331+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:37.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:37.331+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:37.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:37.133+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:37.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:37.133+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:37.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:37.133+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:37.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:37.133+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:38.311 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:38.184+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:38.311 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:38.184+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:38.311 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:38.184+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:38.312 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:38.185+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:38.312 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:38.123+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:38.312 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:38.123+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:38.312 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:38.123+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:38.312 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:38.124+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:38.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:38.320+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:38.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:38.320+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:38.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:38.320+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:38.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:38.321+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:39.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:39.198+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:39.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:39.198+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:39.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:39.198+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:39.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:39.199+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:39.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:39.163+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:39.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:39.163+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:39.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:39.163+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:39.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:39.164+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:39.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:39.284+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:39.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:39.284+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:39.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:39.284+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:39.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:39.284+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:40.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:40.198+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:40.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:40.198+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:40.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:40.198+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:40.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:40.199+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:40.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:40.214+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:40.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:40.214+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:40.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:40.214+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:40.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:40.215+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:40.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:40.240+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:40.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:40.240+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:40.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:40.240+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:40.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:40.241+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:41.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:41.171+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:41.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:41.171+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:41.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:41.171+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:41.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:41.172+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:41.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:41.228+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:41.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:41.228+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:41.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:41.228+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:41.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:41.229+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:41.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:41.243+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:41.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:41.243+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:41.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:41.243+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:41.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:41.244+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:41.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:02:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:02:41.415+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 723 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:02:42.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:42.161+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:42.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:42.161+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:42.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:42.161+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:42.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:42.162+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:42.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:42.208+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:42.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:42.208+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:42.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:42.208+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:42.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:42.208+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:42.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:42.227+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:42.660 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:42.227+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:42.660 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:42.227+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:42.660 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:42.228+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:43.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:43.133+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:43.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:43.133+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:43.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:43.133+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:43.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:43.134+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:43.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:43.177+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:43.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:43.177+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:43.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:43.177+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:43.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:43.178+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:43.182+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:43.182+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:43.182+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:43.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:43.183+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:44.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:44.156+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:44.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:44.156+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:44.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:44.156+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:44.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:44.157+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:44.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:44.138+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:44.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:44.138+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:44.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:44.138+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:44.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:44.139+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:44.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:44.193+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:44.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:44.193+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:44.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:44.193+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:44.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:44.194+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:45.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:45.181+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:45.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:45.181+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:45.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:45.181+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:45.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:45.182+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:45.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:45.184+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:45.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:45.184+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:45.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:45.184+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:45.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:45.185+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:45.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:45.148+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:45.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:45.148+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:45.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:45.148+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:45.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:45.148+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:46.543 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:46.204+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:46.543 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:46.204+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:46.543 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:46.204+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:46.543 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:46.205+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:46.544 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:02:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:02:46.415+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 750 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:02:46.544 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:46.233+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:46.544 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:46.233+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:46.544 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:46.233+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:46.544 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:46.234+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:46.544 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:46.192+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:46.545 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:46.192+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:46.545 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:46.192+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:46.545 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:46.192+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:47.247 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:47.147+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:47.247 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:47.147+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:47.247 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:47.147+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:47.247 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:47.148+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:47.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:47.253+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:47.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:47.253+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:47.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:47.253+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:47.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:47.254+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:47.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:47.252+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:47.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:47.252+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:47.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:47.252+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:47.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:47.253+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:48.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:48.298+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:48.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:48.298+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:48.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:48.298+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:48.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:48.298+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:48.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:48.278+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:48.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:48.278+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:48.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:48.278+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:48.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:48.279+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:48.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:48.141+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:48.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:48.141+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:48.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:48.141+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:48.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:48.142+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:49.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:49.245+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:49.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:49.245+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:49.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:49.245+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:49.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:49.246+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:49.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:49.310+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:49.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:49.310+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:49.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:49.310+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:49.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:49.311+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:49.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:49.117+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:49.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:49.117+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:49.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:49.117+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:49.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:49.117+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:50.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:50.330+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:50.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:50.330+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:50.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:50.330+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:50.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:50.332+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:50.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:50.248+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:50.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:50.248+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:50.410 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:50.248+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:50.410 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:50.249+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:50.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:50.128+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:50.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:50.128+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:50.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:50.128+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:50.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:50.129+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:51.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:51.296+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:51.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:51.296+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:51.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:51.296+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:51.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:51.297+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:51.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:51.263+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:51.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:51.263+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:51.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:51.263+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:51.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:51.264+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:51.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:51.144+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:51.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:51.144+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:51.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:51.144+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:51.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:51.144+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:51.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:02:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:02:51.415+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 781 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:02:52.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:52.330+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:52.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:52.330+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:52.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:52.330+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:52.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:52.331+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:52.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:52.285+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:52.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:52.285+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:52.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:52.285+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:52.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:52.286+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:52.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:52.183+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:52.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:52.183+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:52.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:52.184+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:52.660 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:52.184+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:53.498 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:53.365+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:53.498 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:53.365+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:53.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:53.365+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:53.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:53.366+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:53.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:53.199+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:53.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:53.199+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:53.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:53.199+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:53.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:53.200+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:53.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:53.284+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:53.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:53.284+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:53.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:53.284+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:53.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:53.284+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:54.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:54.333+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:54.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:54.333+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:54.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:54.333+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:54.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:54.334+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:54.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:54.303+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:54.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:54.303+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:54.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:54.303+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:54.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:54.303+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:54.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:54.234+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:54.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:54.234+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:54.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:54.234+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:54.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:54.235+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:55.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:55.347+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:55.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:55.347+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:55.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:55.347+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:55.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:55.348+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:55.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:55.267+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:55.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:55.267+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:55.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:55.267+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:55.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:55.268+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:55.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:55.213+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:55.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:55.213+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:55.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:55.213+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:55.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:55.214+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:56.431 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:02:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:02:56.416+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 809 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:02:56.431 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:56.171+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:56.431 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:56.171+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:56.431 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:56.171+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:56.431 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:56.172+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:56.432 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:56.276+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:56.432 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:56.276+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:56.432 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:56.276+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:56.432 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:56.277+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:56.432 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:56.368+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:56.432 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:56.368+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:56.432 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:56.368+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:56.432 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:56.369+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:57.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:57.199+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:57.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:57.199+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:57.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:57.199+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:57.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:57.200+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:57.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:57.326+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:57.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:57.326+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:57.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:57.326+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:57.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:57.327+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:57.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:57.383+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:57.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:57.383+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:57.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:57.383+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:57.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:57.384+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:58.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:58.352+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:58.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:58.352+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:58.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:58.352+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:58.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:58.353+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:58.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:58.319+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:58.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:58.319+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:58.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:58.319+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:58.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:58.320+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:58.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:58.215+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:58.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:58.215+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:58.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:58.215+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:58.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:58.216+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:59.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:59.368+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:59.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:59.368+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:59.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:59.368+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:02:59.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:02:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:02:59.369+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:02:59.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:59.277+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:59.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:59.277+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:59.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:59.277+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:02:59.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:02:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:02:59.277+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:02:59.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:59.210+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:59.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:59.210+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:59.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:59.210+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:02:59.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:02:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:02:59.211+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:00.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:00.260+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:00.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:00.260+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:00.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:00.260+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:00.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:00.261+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:00.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:00.400+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:00.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:00.400+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:00.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:00.400+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:00.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:00.401+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:00.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:00.298+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:00.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:00.298+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:00.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:00.298+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:00.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:00.299+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:01.642 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:03:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:03:01.416+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 852 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:03:01.642 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:01.313+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:01.643 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:01.313+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:01.643 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:01.313+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:01.643 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:01.314+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:01.643 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:01.443+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:01.643 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:01.443+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:01.643 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:01.443+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:01.643 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:01.443+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:01.644 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:01.294+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:01.644 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:01.294+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:01.644 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:01.294+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:01.644 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:01.295+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:02.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:02.436+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:02.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:02.436+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:02.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:02.436+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:02.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:02.437+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:02.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:02.285+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:02.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:02.285+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:02.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:02.285+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:02.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:02.286+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:02.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:02.271+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:02.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:02.272+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:02.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:02.272+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:02.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:02.273+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:03.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:03.469+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:03.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:03.469+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:03.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:03.469+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:03.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:03.470+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:03.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:03.325+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:03.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:03.325+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:03.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:03.325+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:03.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:03.326+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:03.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:03.243+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:03.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:03.243+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:03.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:03.243+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:03.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:03.244+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:04.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:04.469+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:04.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:04.469+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:04.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:04.469+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:04.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:04.470+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:04.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:04.352+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:04.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:04.352+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:04.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:04.352+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:04.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:04.352+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:04.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:04.282+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:04.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:04.282+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:04.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:04.282+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:04.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:04.283+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:05.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:05.392+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:05.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:05.392+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:05.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:05.392+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:05.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:05.392+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:05.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:05.474+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:05.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:05.474+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:05.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:05.474+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:05.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:05.475+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:05.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:05.298+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:05.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:05.298+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:05.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:05.298+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:05.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:05.299+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:06.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:03:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:03:06.417+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 879 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:03:06.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:06.501+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:06.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:06.501+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:06.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:06.501+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:06.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:06.502+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:06.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:06.412+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:06.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:06.412+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:06.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:06.412+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:06.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:06.413+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:06.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:06.305+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:06.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:06.305+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:06.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:06.305+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:06.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:06.306+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:07.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:07.519+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:07.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:07.519+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:07.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:07.519+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:07.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:07.520+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:07.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:07.284+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:07.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:07.284+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:07.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:07.284+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:07.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:07.285+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:07.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:07.375+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:07.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:07.375+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:07.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:07.375+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:07.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:07.376+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:08.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:08.558+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:08.660 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:08.558+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:08.660 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:08.558+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:08.660 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:08.559+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:08.660 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:08.391+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:08.660 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:08.391+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:08.660 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:08.391+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:08.660 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:08.392+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:08.661 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:08.330+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:08.661 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:08.330+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:08.661 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:08.330+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:08.661 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:08.331+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:09.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:09.533+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:09.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:09.533+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:09.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:09.533+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:09.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:09.534+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:09.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:09.403+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:09.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:09.403+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:09.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:09.403+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:09.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:09.404+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:09.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:09.367+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:09.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:09.367+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:09.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:09.367+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:09.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:09.368+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:10.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:10.373+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:10.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:10.373+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:10.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:10.373+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:10.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:10.374+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:10.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:10.540+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:10.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:10.540+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:10.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:10.540+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:10.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:10.541+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:10.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:10.395+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:10.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:10.395+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:10.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:10.395+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:10.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:10.396+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:11.643 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:03:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:03:11.417+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 908 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:03:11.644 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:11.522+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:11.644 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:11.522+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:11.644 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:11.522+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:11.644 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:11.523+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:11.644 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:11.325+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:11.644 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:11.325+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:11.644 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:11.325+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:11.644 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:11.326+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:11.645 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:11.410+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:11.645 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:11.410+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:11.645 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:11.410+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:11.645 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:11.411+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:12.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:12.517+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:12.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:12.517+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:12.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:12.517+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:12.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:12.519+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:12.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:12.449+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:12.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:12.449+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:12.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:12.449+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:12.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:12.450+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:12.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:12.346+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:12.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:12.346+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:12.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:12.346+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:12.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:12.346+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:13.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:13.553+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:13.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:13.553+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:13.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:13.553+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:13.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:13.554+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:13.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:13.412+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:13.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:13.412+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:13.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:13.412+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:13.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:13.413+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:13.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:13.331+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:13.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:13.331+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:13.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:13.331+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:13.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:13.332+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:14.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:14.509+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:14.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:14.509+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:14.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:14.509+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:14.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:14.509+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:14.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:14.382+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:14.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:14.382+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:14.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:14.382+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:14.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:14.383+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:14.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:14.436+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:14.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:14.436+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:14.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:14.436+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:14.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:14.437+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:15.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:15.456+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:15.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:15.456+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:15.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:15.456+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:15.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:15.457+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:15.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:15.517+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:15.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:15.518+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:15.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:15.518+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:15.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:15.518+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:15.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:15.405+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:15.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:15.405+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:15.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:15.405+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:15.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:15.406+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:16.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:03:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:03:16.417+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 936 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:03:16.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:16.478+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:16.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:16.478+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:16.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:16.478+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:16.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:16.479+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:16.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:16.460+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:16.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:16.460+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:16.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:16.460+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:16.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:16.461+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:16.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:16.415+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:16.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:16.415+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:16.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:16.415+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:16.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:16.417+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:17.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:17.456+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:17.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:17.456+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:17.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:17.456+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:17.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:17.457+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:17.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:17.451+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:17.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:17.451+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:17.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:17.451+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:17.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:17.452+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:17.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:17.375+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:17.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:17.375+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:17.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:17.375+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:17.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:17.375+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:18.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:18.406+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:18.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:18.406+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:18.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:18.406+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:18.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:18.407+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:18.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:18.413+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:18.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:18.413+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:18.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:18.413+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:18.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:18.413+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:18.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:18.452+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:18.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:18.452+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:18.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:18.452+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:18.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:18.452+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:19.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:19.404+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:19.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:19.404+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:19.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:19.404+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:19.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:19.404+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:19.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:19.403+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:19.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:19.403+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:19.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:19.403+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:19.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:19.404+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:19.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:19.382+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:19.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:19.382+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:19.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:19.382+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:19.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:19.382+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:20.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:20.447+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:20.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:20.447+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:20.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:20.447+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:20.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:20.448+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:20.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:20.365+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:20.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:20.365+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:20.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:20.365+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:20.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:20.366+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:20.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:20.360+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:20.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:20.360+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:20.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:20.360+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:20.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:20.360+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:21.643 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:03:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:03:21.418+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 964 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:03:21.644 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:21.466+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:21.644 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:21.466+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:21.644 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:21.466+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:21.644 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:21.466+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:21.644 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:21.411+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:21.645 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:21.411+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:21.645 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:21.411+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:21.645 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:21.412+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:21.645 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:21.312+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:21.645 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:21.312+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:21.645 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:21.312+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:21.645 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:21.313+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:22.575 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:22.461+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:22.575 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:22.461+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:22.575 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:22.461+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:22.575 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:22.462+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:22.575 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:22.311+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:22.576 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:22.311+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:22.576 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:22.311+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:22.576 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:22.312+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:22.576 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:22.404+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:22.576 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:22.404+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:22.576 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:22.404+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:22.576 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:22.404+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:23.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:23.426+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:23.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:23.426+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:23.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:23.426+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:23.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:23.426+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:23.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:23.317+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:23.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:23.317+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:23.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:23.317+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:23.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:23.318+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:23.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:23.400+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:23.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:23.400+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:23.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:23.400+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:23.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:23.401+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:24.375 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:24.291+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:24.376 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:24.291+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:24.376 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:24.291+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:24.376 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:24.292+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:24.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:24.444+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:24.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:24.444+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:24.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:24.444+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:24.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:24.445+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:24.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:24.371+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:24.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:24.371+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:24.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:24.371+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:24.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:24.372+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:25.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:25.252+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:25.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:25.252+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:25.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:25.252+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:25.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:25.253+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:25.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:25.416+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:25.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:25.416+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:25.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:25.416+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:25.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:25.417+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:25.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:25.349+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:25.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:25.349+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:25.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:25.349+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:25.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:25.350+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:26.547 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:03:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:03:26.419+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 991 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:03:26.548 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:26.397+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:26.548 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:26.397+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:26.548 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:26.397+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:26.548 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:26.398+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:26.548 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:26.261+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:26.548 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:26.261+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:26.548 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:26.261+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:26.549 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:26.262+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:26.549 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:26.386+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:26.549 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:26.386+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:26.549 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:26.386+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:26.549 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:26.387+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:27.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:27.370+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:27.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:27.370+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:27.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:27.370+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:27.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:27.371+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:27.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:27.241+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:27.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:27.241+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:27.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:27.241+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:27.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:27.242+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:27.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:27.350+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:27.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:27.350+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:27.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:27.350+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:27.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:27.351+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:28.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:28.399+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:28.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:28.399+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:28.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:28.399+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:28.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:28.400+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:28.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:28.263+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:28.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:28.263+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:28.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:28.263+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:28.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:28.264+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:28.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:28.342+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:28.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:28.342+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:28.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:28.342+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:28.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:28.344+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:29.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:29.443+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:29.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:29.443+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:29.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:29.443+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:29.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:29.444+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:29.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:29.297+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:29.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:29.297+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:29.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:29.297+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:29.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:29.298+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:29.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:29.375+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:29.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:29.375+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:29.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:29.375+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:29.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:29.376+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:30.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:30.442+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:30.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:30.442+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:30.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:30.442+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:30.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:30.443+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:30.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:30.341+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:30.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:30.341+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:30.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:30.341+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:30.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:30.342+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:30.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:30.373+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:30.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:30.373+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:30.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:30.373+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:30.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:30.374+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:31.645 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:03:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:03:31.419+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 1030 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:03:31.645 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:31.397+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:31.645 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:31.397+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:31.645 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:31.397+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:31.645 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:31.398+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:31.646 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:31.384+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:31.646 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:31.384+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:31.646 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:31.384+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:31.646 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:31.384+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:31.646 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:31.367+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:31.646 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:31.367+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:31.647 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:31.367+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:31.647 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:31.368+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:32.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:32.420+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:32.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:32.420+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:32.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:32.420+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:32.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:32.420+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:32.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:32.400+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:32.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:32.400+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:32.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:32.400+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:32.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:32.401+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:32.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:32.411+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:32.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:32.411+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:32.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:32.411+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:32.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:32.412+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:33.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:33.427+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:33.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:33.427+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:33.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:33.427+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:33.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:33.428+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:33.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:33.388+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:33.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:33.388+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:33.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:33.388+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:33.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:33.388+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:33.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:33.448+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:33.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:33.448+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:33.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:33.448+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:33.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:33.450+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:34.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:34.403+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:34.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:34.403+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:34.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:34.403+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:34.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:34.404+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:34.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:34.460+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:34.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:34.460+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:34.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:34.460+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:34.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:34.461+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:34.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:34.434+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:34.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:34.434+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:34.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:34.434+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:34.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:34.435+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:35.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:35.411+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:35.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:35.411+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:35.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:35.411+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:35.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:35.412+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:35.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:35.444+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:35.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:35.444+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:35.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:35.444+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:35.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:35.444+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:35.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:35.388+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:35.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:35.388+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:35.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:35.388+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:35.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:35.389+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:36.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:03:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:03:36.420+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 1074 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:03:36.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:36.382+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:36.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:36.382+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:36.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:36.382+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:36.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:36.382+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:36.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:36.434+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:36.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:36.434+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:36.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:36.434+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:36.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:36.435+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:36.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:36.453+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:36.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:36.454+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:36.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:36.454+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:36.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:36.455+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:36.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T22:03:36.879+0000 7f8a9cadd700 0 monclient(hunting): authenticate timed out after 300 2024-02-09T22:03:36.884 INFO:teuthology.orchestra.run.smithi006.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-02-09T22:03:37.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:37.405+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:37.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:37.405+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:37.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:37.405+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:37.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:37.405+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:37.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:37.408+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:37.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:37.408+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:37.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:37.408+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:37.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:37.409+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:37.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:37.382+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:37.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:37.382+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:37.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:37.382+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:37.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:37.383+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:37.686 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-02-09T22:03:37.957 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T22:03:38.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:38.408+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:38.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:38.408+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:38.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:38.408+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:38.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:38.409+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:38.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:38.397+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:38.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:38.397+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:38.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:38.397+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:38.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:38.397+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:38.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:38.399+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:38.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:38.399+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:38.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:38.399+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:38.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:38.399+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:39.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:39.444+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:39.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:39.444+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:39.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:39.444+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:39.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:39.445+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:39.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:39.369+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:39.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:39.369+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:39.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:39.369+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:39.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:39.370+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:39.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:39.381+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:39.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:39.381+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:39.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:39.381+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:39.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:39.381+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:40.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:40.413+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:40.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:40.414+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:40.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:40.414+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:40.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:40.415+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:40.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:40.401+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:40.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:40.401+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:40.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:40.401+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:40.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:40.402+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:40.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:40.346+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:40.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:40.346+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:40.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:40.346+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:40.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:40.346+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:41.646 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:03:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:03:41.420+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 1118 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:03:41.646 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:41.337+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:41.646 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:41.337+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:41.646 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:41.337+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:41.646 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:41.338+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:41.646 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:41.379+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:41.647 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:41.379+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:41.647 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:41.379+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:41.647 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:41.380+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:41.647 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:41.419+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:41.647 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:41.419+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:41.647 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:41.419+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:41.647 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:41.420+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:42.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:42.349+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:42.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:42.349+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:42.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:42.349+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:42.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:42.349+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:42.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:42.358+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:42.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:42.358+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:42.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:42.358+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:42.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:42.359+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:42.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:42.400+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:42.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:42.400+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:42.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:42.400+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:42.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:42.401+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:43.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:43.338+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:43.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:43.338+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:43.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:43.338+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:43.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:43.340+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:43.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:43.396+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:43.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:43.396+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:43.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:43.396+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:43.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:43.397+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:43.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:43.438+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:43.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:43.438+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:43.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:43.438+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:43.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:43.438+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:44.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:44.417+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:44.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:44.417+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:44.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:44.417+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:44.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:44.418+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:44.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:44.470+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:44.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:44.470+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:44.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:44.470+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:44.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:44.471+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:44.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:44.379+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:44.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:44.379+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:44.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:44.379+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:44.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:44.380+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:45.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:45.462+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:45.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:45.462+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:45.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:45.462+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:45.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:45.462+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:45.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:45.368+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:45.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:45.368+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:45.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:45.368+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:45.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:45.369+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:45.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:45.475+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:45.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:45.475+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:45.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:45.475+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:45.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:45.476+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:46.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:03:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:03:46.421+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 1161 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:03:46.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:46.447+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:46.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:46.447+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:46.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:46.447+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:46.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:46.448+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:46.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:46.326+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:46.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:46.326+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:46.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:46.326+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:46.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:46.327+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:46.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:46.504+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:46.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:46.504+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:46.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:46.504+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:46.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:46.505+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:47.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:47.450+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:47.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:47.450+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:47.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:47.450+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:47.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:47.451+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:47.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:47.330+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:47.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:47.330+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:47.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:47.330+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:47.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:47.330+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:47.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:47.552+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:47.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:47.552+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:47.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:47.552+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:47.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:47.553+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:48.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:48.483+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:48.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:48.483+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:48.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:48.483+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:48.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:48.484+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:48.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:48.341+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:48.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:48.341+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:48.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:48.341+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:48.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:48.342+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:48.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:48.543+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:48.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:48.543+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:48.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:48.543+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:48.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:48.544+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:49.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:49.513+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:49.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:49.513+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:49.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:49.513+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:49.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:49.514+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:49.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:49.325+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:49.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:49.325+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:49.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:49.325+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:49.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:49.326+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:49.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:49.535+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:49.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:49.535+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:49.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:49.535+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:49.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:49.536+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:50.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:50.520+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:50.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:50.520+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:50.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:50.520+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:50.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:50.521+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:50.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:50.330+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:50.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:50.330+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:50.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:50.330+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:50.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:50.331+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:50.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:50.557+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:50.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:50.557+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:50.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:50.557+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:50.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:50.558+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:51.647 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:51.521+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:51.647 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:51.521+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:51.647 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:51.521+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:51.647 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:51.522+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:51.647 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:03:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:03:51.421+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 1206 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:03:51.648 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:51.530+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:51.648 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:51.530+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:51.648 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:51.530+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:51.648 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:51.530+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:51.648 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:51.378+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:51.648 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:51.378+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:51.648 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:51.378+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:51.649 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:51.379+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:52.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:52.550+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:52.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:52.550+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:52.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:52.550+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:52.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:52.550+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:52.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:52.361+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:52.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:52.361+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:52.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:52.361+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:52.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:52.362+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:52.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:52.548+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:52.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:52.548+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:52.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:52.548+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:52.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:52.549+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:53.541 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:53.522+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:53.542 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:53.522+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:53.542 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:53.522+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:53.542 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:53.522+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:53.542 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:53.403+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:53.542 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:53.403+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:53.542 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:53.403+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:53.542 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:53.404+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:53.542 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:53.535+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:53.543 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:53.535+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:53.543 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:53.535+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:53.543 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:53.536+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:54.830 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:54.536+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:54.830 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:54.536+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:54.830 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:54.536+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:54.830 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:54.536+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:54.830 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:54.545+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:54.831 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:54.545+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:54.831 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:54.545+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:54.831 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:54.546+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:54.831 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:54.404+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:54.831 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:54.404+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:54.831 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:54.404+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:54.831 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:54.404+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:55.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:55.530+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:55.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:55.530+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:55.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:55.530+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:55.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:55.531+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:55.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:55.412+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:55.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:55.412+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:55.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:55.412+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:55.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:55.413+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:55.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:55.551+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:55.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:55.551+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:55.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:55.551+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:55.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:55.552+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:56.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:03:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:03:56.422+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 1250 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:03:56.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:56.573+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:56.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:56.573+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:56.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:56.573+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:56.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:56.574+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:56.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:56.580+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:56.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:56.581+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:56.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:56.581+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:56.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:56.581+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:56.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:56.452+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:56.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:56.452+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:56.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:56.452+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:56.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:56.452+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:57.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:57.558+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:57.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:57.558+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:57.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:57.558+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:57.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:57.558+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:57.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:57.487+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:57.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:57.487+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:57.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:57.487+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:57.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:57.488+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:57.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:57.607+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:57.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:57.607+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:57.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:57.607+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:57.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:57.608+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:58.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:58.512+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:58.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:58.512+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:58.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:58.512+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:58.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:58.512+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:58.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:58.591+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:58.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:58.591+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:58.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:58.591+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:58.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:58.592+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:58.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:58.498+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:58.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:58.498+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:58.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:58.498+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:58.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:58.499+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:59.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:59.532+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:59.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:59.532+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:59.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:59.532+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:03:59.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:03:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:03:59.532+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:03:59.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:59.471+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:59.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:59.471+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:59.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:59.471+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:03:59.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:03:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:03:59.472+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:03:59.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:59.598+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:59.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:59.598+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:59.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:59.598+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:03:59.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:03:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:03:59.599+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:00.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:00.486+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:00.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:00.486+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:00.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:00.486+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:00.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:00.486+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:00.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:00.482+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:00.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:00.482+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:00.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:00.482+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:00.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:00.483+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:00.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:00.638+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:00.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:00.638+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:00.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:00.638+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:00.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:00.639+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:01.690 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:04:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:04:01.423+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 1294 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:04:01.691 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:01.516+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:01.691 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:01.516+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:01.691 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:01.516+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:01.691 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:01.517+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:01.691 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:01.495+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:01.691 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:01.495+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:01.691 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:01.495+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:01.691 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:01.496+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:02.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:01.686+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:02.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:01.686+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:02.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:01.686+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:02.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:01.687+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:02.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:02.644+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:02.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:02.644+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:02.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:02.644+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:02.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:02.645+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:02.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:02.467+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:02.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:02.467+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:02.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:02.467+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:02.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:02.468+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:02.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:02.533+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:02.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:02.533+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:02.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:02.533+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:02.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:02.534+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:03.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:03.578+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:03.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:03.578+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:03.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:03.578+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:03.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:03.578+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:03.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:03.622+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:03.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:03.622+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:03.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:03.622+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:03.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:03.622+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:03.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:03.486+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:03.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:03.486+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:03.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:03.486+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:03.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:03.486+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:04.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:04.484+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:04.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:04.484+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:04.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:04.484+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:04.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:04.485+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:04.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:04.602+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:04.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:04.602+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:04.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:04.602+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:04.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:04.603+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:04.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:04.596+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:04.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:04.596+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:04.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:04.596+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:04.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:04.597+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:05.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:05.609+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:05.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:05.609+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:05.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:05.609+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:05.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:05.610+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:05.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:05.491+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:05.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:05.491+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:05.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:05.491+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:05.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:05.492+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:05.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:05.581+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:05.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:05.581+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:05.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:05.581+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:05.910 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:05.582+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:06.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:04:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:04:06.423+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 1334 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:04:06.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:06.585+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:06.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:06.585+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:06.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:06.585+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:06.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:06.586+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:06.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:06.502+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:06.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:06.502+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:06.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:06.502+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:06.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:06.503+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:06.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:06.630+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:06.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:06.630+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:06.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:06.630+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:06.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:06.631+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:07.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:07.605+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:07.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:07.605+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:07.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:07.605+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:07.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:07.606+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:07.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:07.546+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:07.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:07.546+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:07.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:07.546+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:07.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:07.547+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:07.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:07.626+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:07.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:07.627+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:07.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:07.627+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:07.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:07.628+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:08.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:08.543+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:08.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:08.543+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:08.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:08.543+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:08.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:08.544+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:08.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:08.574+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:08.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:08.574+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:08.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:08.574+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:08.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:08.575+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:08.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:08.623+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:08.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:08.623+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:08.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:08.623+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:08.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:08.624+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:09.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:09.614+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:09.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:09.614+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:09.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:09.614+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:09.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:09.615+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:09.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:09.591+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:09.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:09.591+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:09.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:09.591+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:09.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:09.591+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:09.911 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:09.522+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:09.911 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:09.522+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:09.911 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:09.522+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:09.911 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:09.523+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:10.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:10.653+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:10.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:10.653+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:10.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:10.653+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:10.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:10.654+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:10.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:10.606+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:10.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:10.606+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:10.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:10.606+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:10.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:10.607+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:10.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:10.515+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:10.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:10.515+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:10.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:10.515+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:10.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:10.516+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:11.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:04:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:04:11.424+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 1414 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:04:11.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:11.619+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:11.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:11.619+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:11.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:11.619+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:11.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:11.620+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:11.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:11.584+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:11.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:11.584+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:11.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:11.584+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:11.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:11.585+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:11.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:11.491+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:11.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:11.491+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:11.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:11.491+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:11.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:11.491+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:12.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:12.552+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:12.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:12.552+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:12.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:12.552+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:12.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:12.553+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:12.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:12.575+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:12.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:12.575+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:12.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:12.575+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:12.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:12.575+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:12.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:12.534+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:12.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:12.534+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:12.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:12.534+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:12.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:12.535+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:13.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:13.542+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:13.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:13.542+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:13.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:13.542+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:13.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:13.543+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:13.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:13.554+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:13.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:13.554+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:13.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:13.554+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:13.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:13.555+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:13.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:13.562+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:13.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:13.562+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:13.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:13.562+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:13.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:13.563+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:14.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:14.574+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:14.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:14.574+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:14.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:14.574+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:14.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:14.575+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:14.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:14.596+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:14.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:14.596+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:14.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:14.596+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:14.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:14.597+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:14.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:14.567+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:14.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:14.567+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:14.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:14.568+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:14.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:14.569+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:15.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:15.543+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:15.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:15.543+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:15.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:15.543+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:15.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:15.543+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:15.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:15.598+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:15.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:15.598+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:15.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:15.598+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:15.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:15.599+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:15.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:15.585+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:15.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:15.585+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:15.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:15.585+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:15.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:15.587+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:16.611 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:04:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:04:16.424+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 1507 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:04:16.612 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:16.508+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:16.612 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:16.508+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:16.612 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:16.508+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:16.612 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:16.509+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:16.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:16.607+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:16.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:16.607+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:16.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:16.607+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:16.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:16.608+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:16.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:16.632+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:16.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:16.632+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:16.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:16.632+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:16.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:16.633+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:17.809 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:17.526+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:17.809 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:17.526+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:17.809 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:17.526+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:17.810 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:17.527+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:17.810 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:17.637+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:17.810 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:17.637+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:17.810 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:17.637+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:17.810 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:17.638+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:17.810 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:17.617+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:17.810 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:17.617+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:17.810 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:17.617+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:17.811 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:17.618+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:18.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:18.558+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:18.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:18.558+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:18.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:18.558+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:18.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:18.559+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:18.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:18.575+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:18.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:18.575+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:18.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:18.575+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:18.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:18.577+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:18.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:18.612+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:18.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:18.612+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:18.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:18.612+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:18.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:18.613+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:19.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:19.581+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:19.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:19.581+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:19.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:19.581+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:19.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:19.583+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:19.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:19.565+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:19.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:19.565+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:19.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:19.565+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:19.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:19.565+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:19.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:19.602+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:19.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:19.602+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:19.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:19.602+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:19.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:19.603+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:20.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:20.548+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:20.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:20.548+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:20.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:20.548+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:20.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:20.549+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:20.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:20.581+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:20.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:20.581+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:20.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:20.581+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:20.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:20.582+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:20.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:20.583+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:20.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:20.583+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:20.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:20.583+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:20.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:20.584+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:21.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:04:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:04:21.425+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 1602 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:04:21.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:21.551+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:21.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:21.551+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:21.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:21.551+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:21.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:21.552+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:21.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:21.546+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:21.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:21.546+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:21.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:21.546+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:21.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:21.547+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:21.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:21.535+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:21.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:21.535+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:21.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:21.535+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:21.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:21.536+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:22.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:22.566+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:22.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:22.566+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:22.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:22.566+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:22.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:22.567+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:22.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:22.527+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:22.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:22.527+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:22.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:22.527+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:22.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:22.528+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:22.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:22.523+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:22.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:22.523+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:22.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:22.523+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:22.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:22.523+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:23.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:23.555+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:23.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:23.555+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:23.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:23.555+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:23.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:23.555+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:23.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:23.553+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:23.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:23.553+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:23.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:23.553+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:23.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:23.554+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:23.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:23.492+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:23.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:23.492+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:23.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:23.492+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:23.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:23.492+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:24.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:24.489+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:24.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:24.489+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:24.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:24.489+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:24.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:24.489+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:24.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:24.512+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:24.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:24.512+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:24.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:24.512+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:24.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:24.513+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:24.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:24.519+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:24.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:24.519+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:24.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:24.519+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:24.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:24.520+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:25.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:25.524+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:25.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:25.524+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:25.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:25.524+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:25.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:25.525+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:25.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:25.559+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:25.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:25.559+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:25.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:25.559+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:25.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:25.559+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:25.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:25.457+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:25.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:25.457+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:25.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:25.457+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:25.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:25.458+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:26.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:04:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:04:26.426+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 1694 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:04:26.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:26.528+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:26.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:26.528+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:26.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:26.528+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:26.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:26.529+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:26.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:26.516+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:26.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:26.516+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:26.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:26.516+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:26.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:26.517+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:26.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:26.491+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:26.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:26.491+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:26.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:26.491+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:26.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:26.492+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:27.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:27.562+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:27.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:27.562+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:27.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:27.562+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:27.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:27.563+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:27.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:27.503+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:27.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:27.503+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:27.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:27.503+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:27.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:27.504+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:27.910 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:27.500+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:27.910 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:27.500+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:27.910 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:27.500+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:27.910 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:27.501+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:28.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:28.534+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:28.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:28.534+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:28.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:28.534+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:28.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:28.535+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:28.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:28.471+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:28.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:28.471+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:28.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:28.471+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:28.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:28.471+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:28.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:28.484+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:28.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:28.484+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:28.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:28.484+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:28.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:28.485+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:29.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:29.539+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:29.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:29.539+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:29.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:29.539+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:29.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:29.540+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:29.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:29.438+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:29.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:29.438+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:29.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:29.438+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:29.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:29.439+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:29.910 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:29.507+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:29.910 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:29.507+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:29.910 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:29.507+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:29.910 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:29.507+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:30.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:30.446+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:30.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:30.446+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:30.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:30.446+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:30.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:30.447+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:30.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:30.522+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:30.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:30.522+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:30.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:30.522+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:30.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:30.523+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:30.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:30.522+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:30.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:30.522+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:30.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:30.522+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:30.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:30.523+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:31.695 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:04:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:04:31.426+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 1790 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:04:31.695 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:31.536+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:31.695 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:31.536+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:31.695 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:31.536+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:31.695 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:31.537+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:31.696 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:31.423+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:31.696 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:31.423+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:31.696 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:31.423+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:31.696 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:31.425+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:31.696 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:31.503+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:31.696 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:31.503+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:31.696 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:31.503+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:31.696 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:31.503+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:32.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:32.572+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:32.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:32.572+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:32.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:32.572+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:32.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:32.573+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:32.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:32.425+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:32.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:32.425+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:32.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:32.425+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:32.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:32.426+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:32.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:32.499+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:32.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:32.499+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:32.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:32.499+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:32.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:32.500+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:33.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:33.587+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:33.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:33.587+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:33.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:33.587+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:33.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:33.588+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:33.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:33.459+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:33.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:33.459+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:33.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:33.459+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:33.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:33.460+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:33.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:33.417+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:33.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:33.417+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:33.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:33.417+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:33.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:33.418+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:34.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:34.606+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:34.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:34.606+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:34.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:34.606+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:34.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:34.607+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:34.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:34.441+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:34.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:34.441+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:34.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:34.441+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:34.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:34.442+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:34.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:34.383+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:34.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:34.383+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:34.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:34.383+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:34.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:34.384+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:35.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:35.584+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:35.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:35.584+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:35.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:35.584+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:35.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:35.585+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:35.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:35.424+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:35.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:35.424+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:35.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:35.424+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:35.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:35.425+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:35.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:35.428+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:35.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:35.428+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:35.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:35.428+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:35.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:35.428+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:36.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:04:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:04:36.427+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 1883 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:04:36.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:36.603+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:36.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:36.603+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:36.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:36.603+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:36.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:36.604+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:36.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:36.426+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:36.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:36.426+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:36.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:36.426+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:36.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:36.427+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:36.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:36.443+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:36.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:36.443+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:36.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:36.443+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:36.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:36.444+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:37.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:37.579+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:37.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:37.579+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:37.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:37.579+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:37.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:37.580+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:37.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:37.467+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:37.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:37.467+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:37.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:37.467+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:37.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:37.468+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:37.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:37.467+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:37.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:37.467+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:37.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:37.467+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:37.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:37.468+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:38.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:38.559+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:38.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:38.559+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:38.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:38.559+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:38.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:38.560+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:38.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:38.493+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:38.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:38.493+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:38.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:38.493+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:38.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:38.494+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:38.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:38.470+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:38.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:38.470+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:38.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:38.470+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:38.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:38.471+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:39.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:39.571+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:39.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:39.571+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:39.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:39.571+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:39.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:39.572+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:39.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:39.445+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:39.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:39.445+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:39.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:39.445+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:39.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:39.446+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:39.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:39.485+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:39.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:39.485+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:39.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:39.485+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:39.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:39.486+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:40.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:40.546+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:40.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:40.546+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:40.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:40.546+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:40.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:40.547+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:40.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:40.430+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:40.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:40.430+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:40.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:40.430+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:40.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:40.432+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:40.910 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:40.465+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:40.910 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:40.465+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:40.910 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:40.465+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:40.910 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:40.466+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:41.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:04:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:04:41.427+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 1977 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:04:41.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:41.543+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:41.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:41.543+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:41.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:41.543+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:41.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:41.544+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:41.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:41.410+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:41.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:41.410+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:41.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:41.410+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:41.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:41.411+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:41.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:41.440+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:41.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:41.440+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:41.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:41.440+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:41.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:41.441+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:42.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:42.510+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:42.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:42.510+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:42.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:42.510+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:42.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:42.510+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:42.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:42.491+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:42.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:42.491+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:42.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:42.491+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:42.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:42.492+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:42.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:42.436+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:42.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:42.436+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:42.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:42.436+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:42.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:42.437+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:43.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:43.471+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:43.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:43.471+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:43.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:43.471+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:43.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:43.471+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:43.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:43.389+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:43.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:43.389+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:43.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:43.390+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:43.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:43.391+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:43.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:43.455+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:43.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:43.455+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:43.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:43.455+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:43.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:43.455+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:44.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:44.481+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:44.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:44.481+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:44.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:44.481+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:44.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:44.482+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:44.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:44.476+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:44.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:44.476+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:44.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:44.476+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:44.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:44.477+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:44.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:44.427+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:44.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:44.427+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:44.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:44.427+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:44.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:44.428+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:45.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:45.406+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:45.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:45.406+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:45.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:45.406+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:45.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:45.407+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:45.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:45.482+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:45.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:45.482+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:45.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:45.482+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:45.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:45.483+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:45.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:45.430+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:45.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:45.430+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:45.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:45.430+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:45.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:45.431+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:46.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:04:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:04:46.427+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 2068 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:04:46.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:46.388+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:46.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:46.388+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:46.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:46.388+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:46.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:46.389+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:46.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:46.363+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:46.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:46.363+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:46.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:46.363+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:46.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:46.363+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:46.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:46.476+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:46.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:46.476+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:46.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:46.476+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:46.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:46.477+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:47.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:47.460+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:47.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:47.460+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:47.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:47.460+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:47.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:47.461+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:47.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:47.418+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:47.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:47.418+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:47.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:47.418+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:47.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:47.418+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:47.660 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:47.362+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:47.660 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:47.362+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:47.660 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:47.362+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:47.660 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:47.363+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:48.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:48.458+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:48.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:48.458+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:48.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:48.458+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:48.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:48.458+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:48.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:48.372+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:48.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:48.372+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:48.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:48.372+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:48.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:48.372+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:48.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:48.401+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:48.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:48.401+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:48.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:48.401+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:48.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:48.402+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:49.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:49.415+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:49.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:49.415+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:49.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:49.415+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:49.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:49.416+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:49.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:49.402+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:49.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:49.402+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:49.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:49.402+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:49.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:49.403+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:49.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:49.394+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:49.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:49.394+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:49.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:49.394+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:49.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:49.395+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:50.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:50.412+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:50.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:50.412+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:50.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:50.412+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:50.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:50.413+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:50.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:50.378+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:50.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:50.378+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:50.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:50.378+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:50.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:50.379+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:50.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:50.382+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:50.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:50.382+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:50.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:50.382+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:50.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:50.383+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:51.649 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:04:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:04:51.429+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 2163 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:04:51.650 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:51.341+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:51.650 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:51.341+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:51.650 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:51.341+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:51.650 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:51.341+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:51.651 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:51.419+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:51.651 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:51.419+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:51.651 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:51.419+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:51.651 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:51.420+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:51.651 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:51.401+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:51.651 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:51.401+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:51.651 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:51.401+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:51.652 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:51.401+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:52.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:52.375+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:52.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:52.375+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:52.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:52.375+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:52.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:52.376+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:52.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:52.418+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:52.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:52.418+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:52.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:52.418+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:52.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:52.419+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:52.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:52.387+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:52.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:52.387+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:52.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:52.387+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:52.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:52.387+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:53.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:53.385+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:53.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:53.385+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:53.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:53.385+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:53.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:53.386+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:53.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:53.418+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:53.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:53.418+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:53.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:53.418+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:53.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:53.419+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:53.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:53.437+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:53.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:53.437+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:53.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:53.437+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:53.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:53.439+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:54.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:54.377+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:54.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:54.377+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:54.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:54.377+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:54.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:54.377+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:54.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:54.351+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:54.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:54.351+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:54.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:54.351+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:54.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:54.352+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:54.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:54.414+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:54.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:54.414+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:54.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:54.414+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:54.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:54.415+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:55.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:55.395+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:55.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:55.395+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:55.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:55.395+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:55.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:55.396+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:55.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:55.372+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:55.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:55.372+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:55.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:55.372+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:55.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:55.373+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:55.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:55.456+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:55.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:55.456+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:55.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:55.456+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:55.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:55.457+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:56.558 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:04:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:04:56.429+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 2255 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:04:56.559 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:56.334+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:56.559 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:56.334+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:56.559 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:56.334+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:56.559 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:56.335+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:56.559 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:56.439+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:56.559 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:56.439+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:56.559 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:56.439+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:56.559 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:56.440+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:56.560 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:56.377+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:56.560 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:56.377+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:56.560 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:56.377+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:56.560 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:56.377+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:57.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:57.455+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:57.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:57.455+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:57.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:57.455+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:57.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:57.456+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:57.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:57.348+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:57.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:57.348+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:57.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:57.348+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:57.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:57.349+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:57.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:57.384+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:57.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:57.384+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:57.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:57.384+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:57.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:57.385+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:58.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:58.423+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:58.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:58.423+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:58.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:58.423+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:58.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:58.424+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:58.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:58.397+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:58.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:58.397+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:58.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:58.397+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:58.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:58.398+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:58.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:58.395+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:58.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:58.395+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:58.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:58.395+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:58.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:58.395+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:59.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:59.384+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:59.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:59.384+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:59.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:59.384+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:04:59.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:04:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:04:59.384+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:04:59.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:59.399+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:59.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:59.399+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:59.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:59.399+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:04:59.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:04:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:04:59.400+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:04:59.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:59.437+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:59.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:59.437+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:59.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:59.437+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:04:59.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:04:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:04:59.437+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:00.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:00.341+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:00.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:00.341+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:00.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:00.341+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:00.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:00.342+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:00.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:00.422+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:00.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:00.422+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:00.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:00.422+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:00.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:00.423+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:00.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:00.460+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:00.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:00.460+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:00.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:00.460+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:00.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:00.461+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:01.650 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:01.375+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:01.650 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:01.375+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:01.650 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:01.375+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:01.651 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:01.376+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:01.651 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:05:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:05:01.430+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 2349 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:05:01.651 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:01.497+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:01.652 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:01.497+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:01.652 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:01.497+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:01.652 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:01.497+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:01.652 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:01.357+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:01.653 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:01.357+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:01.653 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:01.357+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:01.653 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:01.358+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:02.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:02.366+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:02.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:02.366+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:02.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:02.366+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:02.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:02.367+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:02.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:02.365+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:02.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:02.365+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:02.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:02.365+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:02.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:02.366+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:02.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:02.539+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:02.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:02.539+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:02.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:02.539+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:02.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:02.540+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:03.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:03.552+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:03.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:03.552+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:03.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:03.552+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:03.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:03.553+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:03.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:03.322+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:03.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:03.322+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:03.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:03.322+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:03.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:03.323+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:03.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:03.356+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:03.660 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:03.356+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:03.660 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:03.356+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:03.660 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:03.357+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:04.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:04.563+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:04.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:04.563+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:04.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:04.563+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:04.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:04.564+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:04.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:04.331+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:04.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:04.331+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:04.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:04.331+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:04.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:04.332+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:04.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:04.314+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:04.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:04.314+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:04.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:04.314+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:04.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:04.315+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:05.605 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:05.340+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:05.605 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:05.340+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:05.605 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:05.340+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:05.605 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:05.341+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:05.606 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:05.354+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:05.606 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:05.354+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:05.606 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:05.354+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:05.606 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:05.355+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:05.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:05.600+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:05.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:05.600+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:05.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:05.600+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:05.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:05.601+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:06.605 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:05:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:05:06.430+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 2441 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:05:06.605 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:06.332+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:06.605 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:06.332+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:06.606 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:06.332+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:06.606 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:06.333+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:06.606 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:06.360+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:06.606 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:06.360+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:06.606 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:06.360+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:06.606 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:06.360+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:06.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:06.601+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:06.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:06.601+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:06.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:06.601+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:06.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:06.602+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:07.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:07.358+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:07.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:07.358+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:07.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:07.358+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:07.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:07.359+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:07.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:07.582+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:07.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:07.582+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:07.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:07.582+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:07.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:07.583+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:07.660 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:07.386+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:07.660 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:07.386+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:07.660 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:07.386+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:07.660 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:07.387+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:08.636 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:08.387+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:08.636 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:08.387+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:08.636 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:08.387+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:08.636 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:08.388+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:08.636 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:08.318+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:08.636 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:08.318+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:08.637 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:08.318+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:08.637 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:08.319+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:08.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:08.632+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:08.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:08.632+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:08.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:08.632+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:08.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:08.633+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:09.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:09.283+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:09.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:09.283+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:09.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:09.283+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:09.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:09.284+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:09.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:09.354+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:09.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:09.354+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:09.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:09.354+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:09.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:09.355+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:10.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:09.674+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:10.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:09.674+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:10.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:09.674+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:10.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:09.675+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:10.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:10.388+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:10.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:10.388+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:10.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:10.388+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:10.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:10.388+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:10.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:10.248+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:10.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:10.248+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:10.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:10.248+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:10.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:10.248+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:11.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:10.658+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:11.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:10.658+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:11.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:10.658+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:11.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:10.659+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:11.558 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:05:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:05:11.431+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 2537 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:05:11.559 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:11.256+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:11.559 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:11.256+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:11.559 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:11.256+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:11.559 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:11.257+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:11.559 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:11.392+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:11.559 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:11.392+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:11.559 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:11.392+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:11.560 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:11.393+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:11.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:11.621+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:11.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:11.621+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:11.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:11.621+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:11.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:11.622+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:12.642 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:12.235+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:12.642 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:12.235+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:12.642 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:12.235+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:12.642 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:12.236+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:12.643 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:12.360+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:12.643 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:12.360+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:12.643 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:12.360+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:12.643 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:12.361+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:12.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:12.638+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:12.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:12.638+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:12.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:12.638+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:12.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:12.639+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:13.498 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:13.355+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:13.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:13.355+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:13.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:13.355+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:13.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:13.355+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:13.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:13.188+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:13.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:13.188+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:13.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:13.188+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:13.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:13.189+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:13.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:13.625+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:13.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:13.625+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:13.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:13.625+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:13.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:13.625+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:14.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:14.329+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:14.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:14.329+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:14.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:14.329+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:14.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:14.329+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:14.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:14.227+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:14.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:14.227+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:14.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:14.227+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:14.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:14.228+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:15.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:14.663+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:15.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:14.663+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:15.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:14.663+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:15.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:14.664+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:15.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:15.233+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:15.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:15.233+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:15.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:15.233+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:15.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:15.234+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:15.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:15.325+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:15.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:15.325+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:15.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:15.325+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:15.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:15.326+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:16.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:15.672+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:16.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:15.672+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:16.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:15.672+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:16.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:15.673+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:16.560 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:05:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:05:16.432+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 2630 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:05:16.561 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:16.212+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:16.561 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:16.212+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:16.561 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:16.212+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:16.561 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:16.213+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:16.561 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:16.279+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:16.561 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:16.279+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:16.561 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:16.279+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:16.561 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:16.280+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:16.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:16.712+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:16.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:16.712+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:16.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:16.712+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:16.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:16.713+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:17.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:17.292+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:17.660 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:17.292+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:17.660 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:17.292+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:17.660 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:17.293+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:17.660 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:17.257+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:17.660 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:17.257+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:17.660 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:17.257+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:17.660 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:17.258+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:18.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:17.692+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:18.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:17.692+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:18.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:17.692+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:18.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:17.693+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:18.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:18.317+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:18.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:18.317+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:18.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:18.318+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:18.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:18.319+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:18.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:18.272+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:18.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:18.272+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:18.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:18.272+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:18.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:18.273+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:19.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:18.673+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:19.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:18.673+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:19.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:18.673+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:19.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:18.674+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:19.655 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:19.291+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:19.655 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:19.291+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:19.655 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:19.291+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:19.655 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:19.292+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:19.655 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:19.353+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:19.655 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:19.353+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:19.655 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:19.353+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:19.656 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:19.354+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:19.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:19.651+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:19.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:19.651+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:19.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:19.651+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:19.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:19.652+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:20.611 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:20.282+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:20.611 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:20.282+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:20.611 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:20.282+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:20.611 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:20.283+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:20.611 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:20.324+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:20.611 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:20.324+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:20.611 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:20.324+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:20.611 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:20.325+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:20.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:20.606+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:20.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:20.606+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:20.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:20.606+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:20.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:20.607+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:21.556 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:05:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:05:21.432+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 2726 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:05:21.557 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:21.276+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:21.557 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:21.276+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:21.557 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:21.276+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:21.557 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:21.277+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:21.557 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:21.278+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:21.557 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:21.278+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:21.557 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:21.278+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:21.557 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:21.279+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:21.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:21.613+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:21.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:21.613+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:21.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:21.613+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:21.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:21.614+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:22.579 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:22.246+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:22.579 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:22.246+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:22.580 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:22.246+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:22.580 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:22.247+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:22.580 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:22.243+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:22.580 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:22.243+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:22.580 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:22.243+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:22.580 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:22.243+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:22.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:22.575+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:22.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:22.575+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:22.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:22.575+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:22.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:22.577+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:23.498 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:23.195+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:23.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:23.195+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:23.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:23.195+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:23.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:23.196+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:23.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:23.241+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:23.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:23.241+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:23.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:23.241+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:23.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:23.241+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:23.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:23.564+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:23.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:23.564+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:23.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:23.564+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:23.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:23.565+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:24.213 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:24.162+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:24.214 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:24.162+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:24.214 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:24.162+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:24.214 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:24.163+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:24.544 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:24.209+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:24.545 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:24.209+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:24.545 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:24.209+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:24.545 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:24.210+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:24.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:24.540+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:24.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:24.540+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:24.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:24.540+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:24.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:24.541+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:25.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:25.149+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:25.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:25.149+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:25.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:25.149+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:25.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:25.150+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:25.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:25.181+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:25.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:25.181+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:25.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:25.181+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:25.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:25.182+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:25.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:25.499+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:25.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:25.499+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:25.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:25.499+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:25.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:25.500+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:26.449 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:05:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:05:26.433+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 2818 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:05:26.450 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:26.204+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:26.450 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:26.204+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:26.450 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:26.204+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:26.450 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:26.205+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:26.450 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:26.192+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:26.450 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:26.192+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:26.450 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:26.192+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:26.451 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:26.193+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:26.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:26.460+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:26.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:26.460+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:26.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:26.460+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:26.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:26.461+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:27.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:27.464+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:27.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:27.464+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:27.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:27.464+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:27.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:27.465+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:27.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:27.219+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:27.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:27.219+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:27.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:27.219+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:27.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:27.220+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:27.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:27.218+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:27.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:27.218+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:27.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:27.218+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:27.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:27.219+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:28.508 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:28.186+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:28.508 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:28.186+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:28.508 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:28.186+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:28.508 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:28.187+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:28.508 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:28.221+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:28.508 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:28.221+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:28.508 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:28.221+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:28.508 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:28.222+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:28.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:28.502+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:28.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:28.502+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:28.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:28.502+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:28.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:28.503+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:29.543 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:29.234+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:29.544 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:29.234+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:29.544 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:29.234+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:29.544 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:29.235+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:29.544 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:29.197+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:29.544 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:29.197+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:29.544 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:29.197+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:29.544 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:29.197+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:29.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:29.539+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:29.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:29.539+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:29.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:29.539+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:29.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:29.540+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:30.542 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:30.272+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:30.542 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:30.272+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:30.542 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:30.272+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:30.543 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:30.272+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:30.543 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:30.160+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:30.543 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:30.160+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:30.543 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:30.160+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:30.543 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:30.161+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:30.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:30.538+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:30.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:30.538+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:30.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:30.538+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:30.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:30.539+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:31.438 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:31.287+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:31.438 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:31.287+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:31.438 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:31.287+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:31.438 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:31.288+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:31.438 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:31.179+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:31.439 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:31.179+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:31.439 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:31.179+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:31.439 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:31.180+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:31.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:05:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:05:31.434+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 2913 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:05:31.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:31.537+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:31.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:31.537+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:31.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:31.537+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:31.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:31.538+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:32.550 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:32.305+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:32.550 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:32.305+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:32.550 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:32.305+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:32.550 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:32.306+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:32.550 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:32.167+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:32.550 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:32.167+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:32.550 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:32.167+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:32.551 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:32.168+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:32.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:32.546+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:32.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:32.546+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:32.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:32.546+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:32.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:32.547+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:33.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:33.268+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:33.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:33.268+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:33.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:33.268+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:33.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:33.268+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:33.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:33.121+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:33.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:33.121+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:33.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:33.121+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:33.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:33.122+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:33.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:33.528+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:33.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:33.528+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:33.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:33.528+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:33.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:33.529+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:34.536 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:34.283+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:34.536 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:34.283+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:34.536 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:34.283+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:34.536 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:34.285+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:34.536 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:34.170+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:34.536 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:34.170+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:34.536 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:34.170+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:34.536 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:34.170+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:34.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:34.531+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:34.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:34.531+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:34.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:34.531+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:34.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:34.532+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:35.586 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:35.189+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:35.587 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:35.189+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:35.587 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:35.189+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:35.587 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:35.190+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:35.587 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:35.324+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:35.587 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:35.324+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:35.587 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:35.324+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:35.587 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:35.325+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:35.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:35.582+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:35.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:35.582+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:35.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:35.582+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:35.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:35.583+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:36.462 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:05:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:05:36.434+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 3005 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:05:36.462 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:36.209+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:36.462 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:36.209+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:36.462 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:36.209+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:36.462 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:36.210+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:36.463 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:36.350+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:36.463 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:36.350+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:36.463 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:36.350+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:36.463 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:36.351+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:36.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:36.569+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:36.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:36.569+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:36.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:36.569+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:36.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:36.570+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:37.567 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:37.384+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:37.567 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:37.384+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:37.567 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:37.384+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:37.567 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:37.385+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:37.567 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:37.250+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:37.567 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:37.250+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:37.567 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:37.250+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:37.567 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:37.251+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:37.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:37.562+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:37.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:37.562+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:37.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:37.562+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:37.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:37.563+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:38.607 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:38.424+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:38.607 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:38.424+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:38.607 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:38.424+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:38.607 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:38.424+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:38.607 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:38.287+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:38.607 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:38.287+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:38.607 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:38.287+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:38.607 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:38.287+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:38.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:38.602+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:38.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:38.602+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:38.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:38.602+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:38.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:38.603+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:39.641 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:39.394+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:39.641 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:39.394+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:39.642 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:39.394+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:39.642 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:39.395+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:39.642 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:39.244+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:39.642 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:39.244+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:39.642 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:39.244+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:39.642 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:39.245+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:39.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:39.637+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:39.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:39.637+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:39.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:39.637+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:39.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:39.638+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:40.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:40.355+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:40.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:40.355+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:40.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:40.355+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:40.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:40.356+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:40.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:40.246+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:40.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:40.246+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:40.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:40.246+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:40.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:40.247+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:41.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:40.674+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:41.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:40.674+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:41.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:40.674+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:41.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:40.675+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:41.460 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:05:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:05:41.435+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 3101 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:05:41.461 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:41.317+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:41.461 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:41.317+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:41.461 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:41.317+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:41.461 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:41.318+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:41.461 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:41.199+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:41.462 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:41.199+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:41.462 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:41.199+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:41.462 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:41.199+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:42.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:41.715+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:42.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:41.715+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:42.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:41.715+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:42.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:41.716+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:42.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:42.281+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:42.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:42.281+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:42.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:42.281+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:42.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:42.282+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:42.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:42.214+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:42.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:42.214+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:42.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:42.214+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:42.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:42.215+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:43.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:42.669+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:43.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:42.669+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:43.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:42.669+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:43.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:42.670+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:43.541 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:43.261+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:43.541 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:43.262+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:43.541 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:43.262+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:43.542 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:43.262+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:43.542 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:43.324+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:43.542 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:43.324+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:43.542 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:43.324+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:43.542 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:43.324+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:43.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:43.653+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:43.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:43.653+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:43.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:43.653+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:43.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:43.654+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:44.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:44.263+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:44.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:44.263+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:44.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:44.263+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:44.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:44.265+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:44.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:44.279+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:44.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:44.279+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:44.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:44.279+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:44.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:44.280+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:45.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:44.699+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:45.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:44.699+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:45.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:44.699+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:45.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:44.700+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:45.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:45.291+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:45.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:45.291+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:45.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:45.291+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:45.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:45.291+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:45.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:45.234+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:45.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:45.234+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:45.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:45.234+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:45.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:45.234+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:46.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:45.667+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:46.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:45.667+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:46.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:45.667+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:46.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:45.668+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:46.464 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:05:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:05:46.436+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 3192 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:05:46.464 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:46.249+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:46.464 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:46.249+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:46.464 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:46.249+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:46.464 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:46.250+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:46.464 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:46.209+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:46.465 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:46.209+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:46.465 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:46.209+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:46.465 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:46.209+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:46.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:46.680+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:46.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:46.680+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:46.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:46.680+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:46.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:46.681+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:47.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:47.269+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:47.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:47.269+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:47.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:47.269+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:47.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:47.270+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:47.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:47.205+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:47.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:47.205+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:47.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:47.205+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:47.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:47.206+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:48.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:47.679+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:48.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:47.679+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:48.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:47.679+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:48.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:47.680+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:48.495 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:48.263+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:48.496 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:48.263+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:48.496 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:48.263+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:48.496 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:48.264+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:48.496 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:48.187+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:48.496 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:48.187+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:48.496 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:48.187+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:48.496 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:48.188+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:49.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:48.702+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:49.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:48.702+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:49.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:48.702+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:49.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:48.703+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:49.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:49.232+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:49.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:49.232+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:49.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:49.232+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:49.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:49.232+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:49.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:49.231+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:49.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:49.231+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:49.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:49.231+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:49.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:49.232+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:50.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:49.693+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:50.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:49.693+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:50.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:49.693+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:50.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:49.694+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:50.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:50.206+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:50.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:50.206+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:50.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:50.206+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:50.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:50.207+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:50.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:50.190+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:50.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:50.190+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:50.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:50.190+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:50.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:50.191+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:51.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:50.704+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:51.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:50.704+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:51.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:50.704+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:51.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:50.705+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:51.441 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:51.189+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:51.441 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:51.189+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:51.441 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:51.189+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:51.441 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:51.190+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:51.441 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:51.159+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:51.441 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:51.159+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:51.441 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:51.159+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:51.441 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:51.160+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:51.691 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:05:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:05:51.436+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 3287 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:05:51.692 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:51.686+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:51.692 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:51.686+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:51.692 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:51.686+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:52.141 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:51.687+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:52.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:52.210+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:52.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:52.210+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:52.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:52.210+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:52.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:52.211+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:52.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:52.136+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:52.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:52.137+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:52.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:52.137+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:52.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:52.137+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:53.104 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:52.680+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:53.104 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:52.680+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:53.105 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:52.680+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:53.105 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:52.681+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:53.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:53.100+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:53.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:53.100+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:53.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:53.100+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:53.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:53.101+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:53.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:53.202+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:53.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:53.202+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:53.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:53.202+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:53.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:53.203+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:53.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:53.674+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:53.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:53.674+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:53.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:53.674+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:53.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:53.675+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:54.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:54.143+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:54.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:54.143+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:54.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:54.143+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:54.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:54.144+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:54.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:54.212+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:54.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:54.212+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:54.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:54.212+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:54.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:54.213+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:54.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:54.688+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:54.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:54.688+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:54.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:54.688+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:54.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:54.689+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:55.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:55.207+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:55.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:55.207+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:55.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:55.207+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:55.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:55.208+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:55.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:55.127+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:55.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:55.127+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:55.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:55.127+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:55.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:55.128+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:56.144 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:55.690+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:56.144 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:55.690+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:56.144 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:55.690+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:56.144 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:55.691+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:56.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:56.140+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:56.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:56.140+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:56.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:56.140+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:56.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:56.141+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:56.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:56.175+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:56.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:56.175+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:56.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:56.175+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:56.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:56.176+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:56.720 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:05:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:05:56.437+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 3379 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:05:57.143 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:56.716+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:57.144 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:56.716+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:57.144 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:56.716+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:57.144 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:56.717+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:57.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:57.139+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:57.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:57.139+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:57.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:57.139+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:57.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:57.140+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:57.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:57.174+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:57.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:57.174+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:57.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:57.174+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:57.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:57.175+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:58.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:57.686+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:58.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:57.686+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:58.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:57.686+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:58.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:57.686+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:58.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:58.185+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:58.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:58.185+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:58.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:58.185+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:58.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:58.186+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:58.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:58.156+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:58.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:58.156+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:58.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:58.156+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:58.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:58.157+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:59.111 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:58.674+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:59.111 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:58.674+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:59.112 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:58.674+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:59.112 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:58.675+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:59.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:59.107+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:59.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:59.107+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:59.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:59.107+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:05:59.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:05:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:05:59.108+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:05:59.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:59.148+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:59.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:59.148+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:59.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:59.148+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:05:59.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:05:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:05:59.148+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:05:59.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:59.648+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:59.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:59.648+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:59.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:59.648+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:05:59.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:05:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:05:59.648+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:00.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:00.116+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:00.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:00.116+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:00.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:00.116+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:00.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:00.118+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:00.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:00.082+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:00.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:00.082+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:00.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:00.082+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:00.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:00.083+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:00.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:00.635+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:00.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:00.635+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:00.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:00.635+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:00.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:00.636+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:01.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:01.107+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:01.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:01.107+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:01.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:01.107+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:01.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:01.107+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:01.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:01.037+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:01.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:01.038+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:01.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:01.038+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:01.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:01.039+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:01.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:06:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:06:01.438+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 3475 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:06:01.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:01.684+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:01.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:01.684+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:01.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:01.684+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:01.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:01.685+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:02.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:02.021+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:02.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:02.021+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:02.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:02.021+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:02.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:02.021+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:02.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:02.112+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:02.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:02.112+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:02.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:02.112+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:02.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:02.112+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:02.987 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:02.680+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:02.987 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:02.680+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:02.987 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:02.680+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:02.988 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:02.681+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:03.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:03.110+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:03.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:03.110+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:03.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:03.110+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:03.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:03.110+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:03.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:02.983+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:03.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:02.983+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:03.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:02.983+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:03.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:02.984+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:03.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:03.652+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:03.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:03.652+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:03.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:03.652+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:03.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:03.652+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:04.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:03.964+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:04.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:03.964+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:04.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:03.964+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:04.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:03.965+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:04.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:04.139+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:04.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:04.139+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:04.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:04.139+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:04.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:04.140+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:04.969 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:04.664+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:04.969 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:04.664+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:04.969 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:04.664+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:04.969 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:04.665+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:05.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:05.183+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:05.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:05.183+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:05.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:05.183+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:05.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:05.184+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:05.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:04.965+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:05.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:04.965+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:05.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:04.965+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:05.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:04.965+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:05.921 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:05.662+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:05.921 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:05.662+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:05.921 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:05.662+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:05.921 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:05.663+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:06.214 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:05.917+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:06.214 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:05.917+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:06.214 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:05.917+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:06.214 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:05.918+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:06.467 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:06:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:06:06.439+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 3568 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:06:06.468 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:06.209+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:06.468 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:06.209+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:06.468 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:06.209+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:06.468 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:06.210+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:06.881 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:06.623+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:06.881 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:06.623+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:06.881 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:06.623+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:06.881 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:06.624+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:07.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:06.877+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:07.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:06.877+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:07.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:06.877+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:07.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:06.878+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:07.620 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:07.217+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:07.620 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:07.217+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:07.620 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:07.217+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:07.620 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:07.218+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:07.890 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:07.615+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:07.890 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:07.615+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:07.890 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:07.615+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:07.890 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:07.616+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:08.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:07.886+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:08.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:07.886+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:08.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:07.886+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:08.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:07.887+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:08.624 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:08.220+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:08.625 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:08.220+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:08.625 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:08.220+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:08.625 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:08.221+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:08.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:08.620+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:08.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:08.620+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:08.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:08.620+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:08.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:08.621+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:08.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:08.863+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:08.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:08.863+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:08.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:08.863+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:08.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:08.864+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:09.645 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:09.250+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:09.646 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:09.250+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:09.646 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:09.250+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:09.646 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:09.251+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:09.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:09.641+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:09.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:09.641+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:09.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:09.641+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:09.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:09.642+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:09.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:09.847+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:09.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:09.847+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:09.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:09.847+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:09.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:09.848+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:10.604 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:10.256+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:10.604 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:10.256+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:10.604 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:10.256+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:10.604 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:10.257+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:10.870 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:10.600+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:10.871 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:10.600+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:10.871 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:10.600+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:10.871 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:10.601+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:11.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:10.866+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:11.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:10.866+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:11.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:10.866+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:11.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:10.867+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:11.567 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:11.281+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:11.567 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:11.281+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:11.567 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:11.281+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:11.567 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:11.282+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:11.568 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:06:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:06:11.439+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 3665 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:06:11.881 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:11.563+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:11.882 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:11.563+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:11.882 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:11.563+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:11.882 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:11.564+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:12.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:11.877+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:12.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:11.877+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:12.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:11.877+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:12.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:11.878+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:12.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:12.321+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:12.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:12.321+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:12.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:12.321+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:12.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:12.322+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:12.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:12.553+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:12.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:12.553+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:12.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:12.553+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:12.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:12.554+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:13.310 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:12.919+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:13.310 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:12.919+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:13.310 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:12.919+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:13.310 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:12.920+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:13.573 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:13.305+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:13.573 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:13.305+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:13.573 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:13.305+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:13.573 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:13.306+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:13.906 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:13.568+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:13.906 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:13.568+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:13.906 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:13.568+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:13.906 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:13.570+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:14.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:13.902+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:14.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:13.902+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:14.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:13.902+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:14.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:13.902+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:14.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:14.302+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:14.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:14.302+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:14.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:14.302+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:14.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:14.302+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:14.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:14.546+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:14.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:14.546+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:14.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:14.546+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:14.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:14.546+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:15.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:14.860+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:15.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:14.860+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:15.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:14.860+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:15.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:14.861+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:15.517 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:15.511+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:15.517 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:15.511+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:15.517 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:15.511+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:15.517 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:15.261+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:15.517 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:15.261+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:15.517 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:15.261+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:15.518 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:15.263+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:15.878 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:15.512+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:16.142 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:15.873+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:16.142 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:15.873+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:16.142 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:15.873+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:16.142 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:15.874+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:16.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:16.296+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:16.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:16.296+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:16.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:16.296+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:16.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:16.297+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:16.898 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:06:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:06:16.439+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 3759 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:06:16.898 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:16.503+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:16.898 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:16.503+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:16.898 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:16.503+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:16.898 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:16.505+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:17.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:16.893+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:17.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:16.893+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:17.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:16.893+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:17.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:16.893+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:17.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:17.346+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:17.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:17.346+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:17.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:17.346+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:17.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:17.347+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:17.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:17.495+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:17.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:17.495+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:17.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:17.495+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:17.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:17.496+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:18.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:17.896+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:18.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:17.896+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:18.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:17.896+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:18.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:17.896+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:18.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:18.384+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:18.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:18.384+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:18.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:18.384+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:18.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:18.385+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:18.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:18.511+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:18.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:18.511+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:18.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:18.511+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:18.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:18.512+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:19.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:18.855+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:19.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:18.855+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:19.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:18.855+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:19.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:18.855+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:19.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:19.400+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:19.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:19.400+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:19.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:19.400+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:19.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:19.401+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:19.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:19.492+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:19.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:19.492+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:19.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:19.492+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:19.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:19.493+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:20.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:19.816+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:20.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:19.816+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:20.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:19.816+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:20.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:19.816+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:20.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:20.383+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:20.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:20.383+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:20.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:20.383+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:20.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:20.383+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:20.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:20.524+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:20.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:20.524+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:20.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:20.524+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:20.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:20.525+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:21.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:20.866+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:21.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:20.866+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:21.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:20.866+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:21.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:20.866+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:21.654 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:21.392+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:21.655 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:21.392+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:21.655 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:21.392+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:21.655 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:21.393+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:21.655 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:06:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:06:21.441+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 3855 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:06:21.655 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:21.487+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:21.655 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:21.487+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:21.655 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:21.487+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:21.656 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:21.488+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:21.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:21.878+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:21.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:21.878+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:21.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:21.878+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:21.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:21.879+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:22.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:22.433+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:22.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:22.433+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:22.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:22.433+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:22.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:22.433+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:22.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:22.455+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:22.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:22.455+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:22.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:22.455+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:22.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:22.456+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:23.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:22.895+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:23.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:22.895+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:23.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:22.895+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:23.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:22.896+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:23.883 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:23.407+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:23.883 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:23.407+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:23.883 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:23.407+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:23.883 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:23.408+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:23.883 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:23.427+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:23.883 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:23.427+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:23.883 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:23.427+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:23.884 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:23.428+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:24.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:23.878+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:24.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:23.878+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:24.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:23.878+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:24.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:23.880+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:24.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:24.360+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:24.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:24.360+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:24.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:24.360+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:24.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:24.361+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:24.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:24.391+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:24.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:24.391+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:24.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:24.391+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:24.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:24.391+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:25.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:24.841+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:25.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:24.841+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:25.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:24.841+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:25.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:24.842+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:25.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:25.313+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:25.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:25.313+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:25.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:25.313+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:25.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:25.314+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:25.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:25.389+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:25.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:25.389+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:25.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:25.389+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:25.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:25.389+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:26.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:25.849+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:26.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:25.849+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:26.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:25.849+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:26.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:25.850+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:26.569 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:06:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:06:26.442+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 3947 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:06:26.570 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:26.274+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:26.570 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:26.274+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:26.570 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:26.274+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:26.570 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:26.275+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:26.570 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:26.372+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:26.570 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:26.372+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:26.570 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:26.372+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:26.570 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:26.373+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:27.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:26.836+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:27.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:26.836+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:27.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:26.836+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:27.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:26.837+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:27.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:27.302+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:27.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:27.302+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:27.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:27.302+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:27.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:27.302+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:27.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:27.364+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:27.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:27.364+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:27.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:27.364+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:27.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:27.365+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:28.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:27.887+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:28.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:27.887+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:28.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:27.887+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:28.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:27.888+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:28.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:28.303+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:28.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:28.303+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:28.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:28.303+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:28.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:28.304+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:28.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:28.385+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:28.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:28.385+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:28.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:28.385+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:28.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:28.386+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:29.312 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:28.925+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:29.312 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:28.925+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:29.312 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:28.925+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:29.312 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:28.926+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:29.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:29.306+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:29.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:29.306+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:29.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:29.306+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:29.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:29.308+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:29.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:29.378+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:29.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:29.378+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:29.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:29.378+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:29.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:29.378+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:30.332 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:29.954+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:30.332 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:29.954+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:30.332 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:29.954+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:30.332 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:29.955+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:30.660 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:30.328+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:30.660 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:30.328+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:30.660 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:30.328+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:30.660 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:30.329+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:30.660 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:30.332+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:30.661 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:30.332+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:30.661 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:30.332+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:30.661 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:30.332+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:31.303 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:30.913+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:31.303 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:30.913+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:31.303 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:30.913+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:31.303 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:30.914+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:31.304 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:31.299+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:31.304 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:31.299+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:31.304 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:31.299+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:31.569 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:06:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:06:31.442+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 4045 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:06:31.569 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:31.350+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:31.569 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:31.350+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:31.569 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:31.350+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:31.569 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:31.350+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:31.570 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:31.300+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:32.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:31.866+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:32.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:31.866+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:32.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:31.866+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:32.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:31.866+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:32.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:32.325+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:32.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:32.325+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:32.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:32.325+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:32.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:32.326+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:32.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:32.311+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:32.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:32.311+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:32.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:32.311+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:32.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:32.312+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:33.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:32.877+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:33.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:32.877+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:33.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:32.877+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:33.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:32.878+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:33.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:33.352+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:33.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:33.352+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:33.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:33.352+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:33.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:33.353+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:33.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:33.360+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:33.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:33.360+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:33.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:33.360+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:33.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:33.361+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:34.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:33.840+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:34.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:33.840+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:34.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:33.840+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:34.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:33.840+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:34.575 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:34.308+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:34.575 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:34.308+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:34.575 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:34.308+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:34.575 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:34.310+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:34.575 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:34.330+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:34.575 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:34.330+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:34.575 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:34.330+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:34.576 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:34.332+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:35.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:34.883+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:35.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:34.883+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:35.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:34.883+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:35.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:34.884+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:35.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:35.348+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:35.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:35.348+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:35.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:35.348+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:35.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:35.349+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:35.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:35.319+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:35.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:35.319+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:35.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:35.319+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:35.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:35.319+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:36.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:35.865+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:36.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:35.865+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:36.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:35.865+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:36.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:35.866+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:36.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:06:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:06:36.443+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 4139 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:06:36.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:36.362+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:36.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:36.362+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:36.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:36.362+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:36.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:36.363+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:36.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:36.337+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:36.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:36.337+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:36.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:36.337+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:36.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:36.338+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:37.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:36.854+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:37.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:36.854+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:37.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:36.854+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:37.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:36.855+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:37.647 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:37.364+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:37.647 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:37.364+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:37.647 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:37.364+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:37.647 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:37.365+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:37.647 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:37.351+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:37.647 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:37.351+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:37.647 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:37.351+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:37.647 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:37.352+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:37.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:37.848+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:37.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:37.848+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:37.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:37.848+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:37.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:37.849+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:38.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:38.316+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:38.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:38.316+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:38.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:38.316+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:38.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:38.317+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:38.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:38.355+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:38.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:38.355+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:38.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:38.355+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:38.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:38.356+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:39.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:38.894+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:39.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:38.894+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:39.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:38.894+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:39.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:38.895+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:39.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:39.340+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:39.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:39.340+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:39.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:39.340+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:39.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:39.341+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:39.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:39.381+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:39.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:39.381+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:39.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:39.381+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:39.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:39.382+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:40.355 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:39.934+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:40.355 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:39.934+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:40.355 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:39.934+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:40.355 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:39.935+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:40.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:40.350+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:40.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:40.350+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:40.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:40.350+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:40.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:40.351+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:40.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:40.429+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:40.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:40.429+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:40.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:40.429+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:40.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:40.430+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:41.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:40.932+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:41.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:40.932+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:41.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:40.932+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:41.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:40.933+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:41.655 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:06:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:06:41.444+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 4235 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:06:41.655 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:41.451+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:41.655 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:41.451+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:41.656 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:41.451+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:41.656 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:41.452+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:41.656 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:41.329+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:41.656 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:41.329+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:41.656 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:41.329+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:41.656 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:41.330+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:42.315 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:41.954+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:42.315 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:41.954+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:42.315 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:41.954+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:42.315 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:41.955+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:42.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:42.311+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:42.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:42.311+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:42.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:42.311+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:42.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:42.311+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:42.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:42.474+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:42.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:42.474+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:42.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:42.474+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:42.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:42.475+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:43.339 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:42.948+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:43.339 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:42.948+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:43.339 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:42.948+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:43.339 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:42.949+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:43.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:43.525+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:43.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:43.525+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:43.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:43.525+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:43.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:43.525+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:43.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:43.334+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:43.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:43.334+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:43.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:43.334+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:43.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:43.335+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:44.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:43.964+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:44.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:43.964+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:44.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:43.964+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:44.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:43.965+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:44.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:44.532+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:44.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:44.532+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:44.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:44.532+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:44.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:44.533+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:44.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:44.292+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:44.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:44.292+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:44.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:44.292+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:44.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:44.293+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:45.275 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:44.999+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:45.275 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:44.999+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:45.275 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:44.999+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:45.275 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:45.000+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:45.537 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:45.271+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:45.537 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:45.271+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:45.537 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:45.271+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:45.537 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:45.272+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:45.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:45.533+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:45.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:45.533+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:45.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:45.533+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:45.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:45.533+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:46.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:46.224+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:46.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:46.224+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:46.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:46.224+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:46.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:46.225+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:46.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:46.050+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:46.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:46.050+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:46.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:46.050+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:46.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:46.051+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:46.863 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:06:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:06:46.445+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 4328 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:06:46.863 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:46.501+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:46.863 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:46.501+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:46.863 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:46.501+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:46.863 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:46.502+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:47.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:47.063+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:47.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:47.063+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:47.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:47.063+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:47.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:47.064+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:47.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:47.238+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:47.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:47.238+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:47.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:47.238+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:47.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:47.239+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:47.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:47.529+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:47.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:47.529+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:47.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:47.529+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:47.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:47.530+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:48.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:48.074+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:48.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:48.074+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:48.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:48.074+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:48.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:48.075+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:48.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:48.235+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:48.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:48.235+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:48.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:48.235+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:48.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:48.236+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:48.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:48.580+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:48.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:48.580+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:48.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:48.580+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:48.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:48.581+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:49.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:49.088+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:49.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:49.088+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:49.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:49.088+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:49.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:49.089+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:49.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:49.205+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:49.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:49.205+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:49.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:49.205+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:49.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:49.205+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:49.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:49.579+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:49.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:49.579+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:49.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:49.579+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:49.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:49.580+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:50.208 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:50.062+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:50.208 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:50.062+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:50.208 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:50.062+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:50.208 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:50.063+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:50.584 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:50.204+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:50.584 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:50.204+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:50.584 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:50.204+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:50.584 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:50.205+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:50.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:50.580+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:50.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:50.580+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:50.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:50.580+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:50.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:50.581+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:51.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:51.175+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:51.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:51.175+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:51.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:51.175+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:51.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:51.176+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:51.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:51.017+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:51.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:51.017+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:51.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:51.017+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:51.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:51.018+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:51.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:06:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:06:51.445+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 4427 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:06:51.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:51.537+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:51.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:51.537+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:51.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:51.537+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:51.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:51.538+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:52.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:52.058+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:52.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:52.058+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:52.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:52.058+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:52.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:52.059+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:52.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:52.216+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:52.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:52.216+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:52.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:52.216+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:52.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:52.217+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:52.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:52.524+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:52.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:52.524+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:52.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:52.524+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:52.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:52.525+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:53.262 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:53.053+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:53.262 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:53.053+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:53.262 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:53.053+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:53.263 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:53.054+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:53.532 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:53.258+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:53.532 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:53.258+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:53.532 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:53.258+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:53.532 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:53.259+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:53.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:53.527+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:53.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:53.528+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:53.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:53.528+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:53.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:53.528+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:54.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:54.101+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:54.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:54.101+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:54.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:54.101+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:54.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:54.102+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:54.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:54.291+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:54.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:54.291+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:54.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:54.291+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:54.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:54.292+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:54.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:54.495+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:54.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:54.495+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:54.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:54.495+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:54.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:54.496+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:55.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:55.332+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:55.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:55.332+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:55.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:55.332+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:55.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:55.332+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:55.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:55.109+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:55.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:55.109+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:55.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:55.109+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:55.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:55.109+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:55.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:55.516+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:55.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:55.516+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:55.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:55.516+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:55.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:55.517+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:56.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:56.079+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:56.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:56.079+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:56.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:56.079+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:56.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:56.079+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:56.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:56.321+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:56.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:56.321+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:56.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:56.321+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:56.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:56.322+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:56.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:06:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:06:56.446+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 4519 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:06:56.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:56.558+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:56.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:56.558+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:56.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:56.558+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:56.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:56.559+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:57.306 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:57.049+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:57.306 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:57.049+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:57.306 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:57.049+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:57.306 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:57.050+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:57.581 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:57.302+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:57.581 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:57.302+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:57.581 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:57.302+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:57.581 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:57.303+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:57.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:57.577+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:57.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:57.577+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:57.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:57.577+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:57.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:57.578+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:58.258 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:58.002+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:58.258 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:58.002+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:58.258 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:58.002+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:58.259 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:58.004+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:58.609 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:58.254+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:58.609 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:58.254+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:58.609 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:58.254+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:58.609 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:58.255+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:58.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:58.605+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:58.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:58.605+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:58.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:58.605+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:58.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:58.606+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:59.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:59.006+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:59.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:59.006+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:59.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:59.006+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:06:59.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:06:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:06:59.006+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:06:59.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:59.213+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:59.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:59.213+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:59.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:59.213+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:06:59.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:06:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:06:59.214+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:06:59.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:59.587+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:59.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:59.587+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:59.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:59.587+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:06:59.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:06:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:06:59.589+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:00.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:00.230+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:00.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:00.230+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:00.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:00.230+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:00.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:00.231+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:00.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:00.056+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:00.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:00.056+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:00.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:00.056+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:00.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:00.057+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:00.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:00.584+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:00.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:00.584+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:00.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:00.584+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:00.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:00.585+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:01.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:01.281+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:01.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:01.281+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:01.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:01.281+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:01.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:01.282+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:01.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:01.077+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:01.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:01.077+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:01.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:01.077+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:01.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:01.078+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:01.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:07:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:07:01.447+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 4623 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:07:01.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:01.591+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:01.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:01.591+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:01.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:01.591+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:01.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:01.592+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:02.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:02.113+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:02.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:02.113+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:02.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:02.114+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:02.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:02.114+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:02.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:02.309+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:02.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:02.309+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:02.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:02.309+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:02.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:02.310+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:02.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:02.602+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:02.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:02.602+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:02.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:02.602+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:02.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:02.604+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:03.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:03.135+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:03.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:03.135+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:03.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:03.135+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:03.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:03.136+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:03.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:03.319+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:03.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:03.319+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:03.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:03.319+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:03.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:03.321+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:03.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:03.624+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:03.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:03.624+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:03.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:03.624+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:03.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:03.625+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:04.365+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:04.365+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:04.365+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:04.367+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:04.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:04.123+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:04.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:04.123+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:04.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:04.123+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:04.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:04.123+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:04.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:04.631+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:04.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:04.631+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:04.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:04.631+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:04.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:04.631+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:05.599 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:05.167+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:05.599 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:05.167+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:05.599 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:05.167+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:05.599 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:05.167+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:05.599 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:05.381+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:05.599 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:05.381+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:05.599 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:05.382+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:05.600 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:05.383+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:05.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:05.595+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:05.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:05.595+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:05.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:05.595+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:05.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:05.596+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:06.378 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:06.122+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:06.378 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:06.122+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:06.379 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:06.122+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:06.379 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:06.122+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:06.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:07:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:07:06.448+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 4720 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:07:06.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:06.374+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:06.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:06.374+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:06.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:06.374+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:06.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:06.375+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:06.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:06.584+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:06.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:06.584+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:06.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:06.584+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:06.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:06.585+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:07.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:07.364+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:07.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:07.364+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:07.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:07.364+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:07.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:07.365+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:07.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:07.151+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:07.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:07.151+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:07.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:07.151+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:07.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:07.151+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:07.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:07.580+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:07.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:07.580+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:07.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:07.580+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:07.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:07.581+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:08.378 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:08.102+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:08.378 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:08.102+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:08.378 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:08.102+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:08.378 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:08.103+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:08.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:08.374+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:08.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:08.374+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:08.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:08.374+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:08.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:08.375+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:08.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:08.575+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:08.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:08.575+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:08.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:08.575+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:08.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:08.576+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:09.370 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:09.093+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:09.370 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:09.093+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:09.370 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:09.093+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:09.370 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:09.094+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:09.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:09.561+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:09.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:09.561+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:09.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:09.561+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:09.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:09.562+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:09.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:09.366+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:09.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:09.366+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:09.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:09.366+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:09.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:09.367+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:10.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:10.335+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:10.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:10.335+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:10.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:10.335+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:10.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:10.335+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:10.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:10.092+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:10.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:10.092+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:10.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:10.092+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:10.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:10.093+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:10.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:10.531+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:10.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:10.531+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:10.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:10.531+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:10.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:10.532+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:11.341 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:11.062+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:11.342 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:11.062+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:11.342 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:11.062+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:11.342 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:11.063+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:11.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:07:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:07:11.449+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 4814 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:07:11.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:11.544+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:11.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:11.544+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:11.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:11.544+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:11.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:11.545+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:11.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:11.337+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:11.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:11.337+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:11.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:11.337+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:11.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:11.338+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:12.385 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:12.033+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:12.385 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:12.033+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:12.385 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:12.033+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:12.385 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:12.035+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:12.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:12.381+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:12.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:12.381+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:12.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:12.381+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:12.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:12.382+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:12.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:12.501+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:12.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:12.501+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:12.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:12.501+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:12.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:12.502+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:13.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:12.996+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:13.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:12.996+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:13.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:12.996+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:13.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:12.996+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:13.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:13.412+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:13.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:13.412+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:13.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:13.412+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:13.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:13.413+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:13.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:13.493+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:13.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:13.493+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:13.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:13.493+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:13.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:13.494+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:14.394 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:14.013+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:14.394 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:14.013+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:14.394 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:14.013+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:14.394 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:14.014+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:14.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:14.469+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:14.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:14.469+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:14.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:14.469+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:14.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:14.470+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:14.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:14.390+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:14.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:14.390+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:14.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:14.390+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:14.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:14.391+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:15.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:15.007+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:15.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:15.007+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:15.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:15.007+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:15.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:15.008+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:15.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:15.503+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:15.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:15.503+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:15.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:15.503+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:15.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:15.504+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:15.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:15.426+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:15.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:15.426+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:15.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:15.426+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:15.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:15.427+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:16.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:16.041+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:16.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:16.041+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:16.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:16.041+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:16.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:16.041+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:16.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:07:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:07:16.449+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 4909 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:07:16.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:16.490+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:16.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:16.490+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:16.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:16.490+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:16.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:16.491+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:16.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:16.473+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:16.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:16.473+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:16.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:16.473+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:16.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:16.474+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:17.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:17.078+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:17.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:17.078+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:17.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:17.078+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:17.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:17.079+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:17.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:17.453+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:17.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:17.453+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:17.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:17.453+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:17.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:17.453+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:17.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:17.475+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:17.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:17.475+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:17.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:17.475+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:17.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:17.476+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:18.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:18.080+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:18.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:18.080+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:18.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:18.080+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:18.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:18.081+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:18.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:18.517+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:18.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:18.517+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:18.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:18.517+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:18.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:18.518+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:18.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:18.464+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:18.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:18.464+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:18.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:18.464+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:18.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:18.465+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:19.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:19.061+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:19.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:19.061+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:19.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:19.061+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:19.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:19.062+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:19.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:19.439+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:19.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:19.439+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:19.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:19.439+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:19.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:19.439+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:19.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:19.555+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:19.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:19.555+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:19.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:19.555+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:19.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:19.556+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:20.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:20.036+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:20.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:20.036+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:20.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:20.036+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:20.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:20.037+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:20.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:20.435+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:20.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:20.435+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:20.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:20.435+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:20.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:20.436+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:20.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:20.521+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:20.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:20.521+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:20.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:20.521+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:20.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:20.522+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:21.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:21.060+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:21.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:21.060+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:21.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:21.060+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:21.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:21.060+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:21.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:07:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:07:21.450+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 5017 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:07:21.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:21.461+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:21.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:21.461+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:21.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:21.461+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:21.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:21.462+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:21.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:21.534+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:21.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:21.534+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:21.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:21.534+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:21.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:21.535+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:22.030+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:22.030+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:22.030+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:22.030+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:22.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:22.421+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:22.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:22.421+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:22.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:22.421+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:22.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:22.422+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:22.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:22.526+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:22.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:22.526+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:22.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:22.526+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:22.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:22.527+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:23.390 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:22.994+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:23.390 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:22.995+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:23.390 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:22.995+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:23.390 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:22.995+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:23.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:23.386+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:23.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:23.386+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:23.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:23.386+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:23.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:23.387+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:23.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:23.558+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:23.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:23.558+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:23.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:23.558+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:23.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:23.559+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:24.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:23.995+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:24.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:23.995+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:24.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:23.995+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:24.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:23.996+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:24.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:24.577+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:24.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:24.577+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:24.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:24.577+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:24.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:24.578+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:24.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:24.412+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:24.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:24.412+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:24.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:24.412+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:24.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:24.413+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:25.368 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:24.954+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:25.368 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:24.954+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:25.368 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:24.954+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:25.368 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:24.955+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:25.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:25.533+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:25.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:25.533+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:25.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:25.533+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:25.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:25.534+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:25.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:25.364+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:25.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:25.364+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:25.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:25.364+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:25.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:25.365+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:26.341 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:25.923+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:26.341 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:25.923+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:26.341 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:25.923+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:26.341 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:25.924+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:26.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:07:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:07:26.451+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 5109 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:07:26.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:26.337+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:26.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:26.337+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:26.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:26.337+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:26.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:26.338+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:26.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:26.541+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:26.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:26.541+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:26.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:26.541+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:26.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:26.542+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:27.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:26.886+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:27.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:26.886+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:27.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:26.886+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:27.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:26.886+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:27.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:27.559+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:27.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:27.559+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:27.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:27.559+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:27.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:27.560+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:27.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:27.380+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:27.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:27.380+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:27.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:27.380+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:27.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:27.381+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:28.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:27.856+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:28.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:27.856+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:28.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:27.856+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:28.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:27.857+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:28.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:28.410+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:28.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:28.410+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:28.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:28.410+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:28.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:28.411+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:28.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:28.527+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:28.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:28.527+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:28.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:28.527+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:28.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:28.528+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:29.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:28.905+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:29.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:28.905+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:29.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:28.905+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:29.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:28.906+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:29.886 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:29.420+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:29.886 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:29.420+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:29.886 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:29.420+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:29.886 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:29.421+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:29.887 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:29.532+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:29.887 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:29.532+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:29.887 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:29.532+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:29.887 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:29.532+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:30.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:29.882+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:30.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:29.882+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:30.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:29.882+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:30.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:29.884+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:30.872 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:30.410+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:30.872 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:30.410+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:30.872 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:30.410+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:30.872 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:30.410+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:30.873 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:30.537+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:30.873 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:30.537+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:30.873 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:30.537+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:30.873 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:30.538+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:31.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:30.868+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:31.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:30.868+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:31.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:30.868+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:31.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:30.869+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:31.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:07:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:07:31.452+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 5206 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:07:31.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:31.411+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:31.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:31.411+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:31.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:31.411+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:31.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:31.412+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:31.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:31.512+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:31.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:31.512+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:31.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:31.512+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:31.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:31.513+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:32.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:31.903+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:32.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:31.903+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:32.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:31.903+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:32.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:31.904+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:32.894 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:32.442+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:32.895 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:32.442+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:32.895 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:32.442+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:32.895 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:32.443+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:32.895 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:32.469+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:32.895 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:32.469+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:32.895 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:32.469+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:32.895 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:32.470+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:33.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:32.890+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:33.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:32.890+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:33.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:32.890+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:33.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:32.891+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:33.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:33.463+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:33.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:33.463+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:33.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:33.463+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:33.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:33.463+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:33.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:33.504+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:33.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:33.504+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:33.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:33.504+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:33.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:33.505+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:34.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:33.936+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:34.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:33.936+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:34.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:33.936+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:34.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:33.937+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:34.898 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:34.461+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:34.898 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:34.461+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:34.899 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:34.461+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:34.899 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:34.463+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:34.900 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:34.503+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:34.900 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:34.503+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:34.901 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:34.503+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:34.901 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:34.504+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:35.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:34.893+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:35.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:34.893+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:35.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:34.893+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:35.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:34.894+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:35.893 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:35.506+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:35.893 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:35.506+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:35.894 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:35.506+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:35.894 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:35.507+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:35.894 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:35.527+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:35.894 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:35.527+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:35.894 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:35.527+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:35.894 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:35.528+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:36.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:35.889+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:36.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:35.889+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:36.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:35.889+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:36.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:35.890+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:36.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:07:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:07:36.453+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 5320 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:07:36.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:36.556+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:36.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:36.556+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:36.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:36.556+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:36.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:36.557+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:36.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:36.510+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:36.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:36.510+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:36.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:36.510+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:36.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:36.511+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:37.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:36.934+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:37.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:36.934+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:37.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:36.934+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:37.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:36.935+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:37.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:37.527+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:37.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:37.527+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:37.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:37.527+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:37.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:37.528+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:37.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:37.559+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:37.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:37.559+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:37.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:37.559+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:37.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:37.560+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:38.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:37.973+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:38.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:37.973+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:38.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:37.973+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:38.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:37.974+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:38.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:38.519+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:38.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:38.519+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:38.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:38.519+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:38.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:38.520+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:38.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:38.583+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:38.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:38.583+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:38.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:38.583+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:38.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:38.584+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:39.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:38.941+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:39.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:38.941+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:39.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:38.941+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:39.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:38.943+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:39.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:39.500+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:39.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:39.500+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:39.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:39.500+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:39.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:39.501+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:39.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:39.623+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:39.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:39.623+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:39.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:39.623+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:39.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:39.624+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:40.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:39.911+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:40.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:39.911+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:40.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:39.911+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:40.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:39.912+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:40.891 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:40.527+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:40.891 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:40.527+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:40.891 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:40.527+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:40.891 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:40.528+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:40.891 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:40.614+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:40.892 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:40.614+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:40.892 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:40.614+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:40.892 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:40.615+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:41.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:40.886+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:41.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:40.886+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:41.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:40.886+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:41.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:40.887+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:41.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:07:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:07:41.454+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 5437 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:07:41.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:41.551+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:41.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:41.551+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:41.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:41.551+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:41.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:41.552+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:41.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:41.573+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:41.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:41.573+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:41.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:41.573+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:41.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:41.574+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:42.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:41.906+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:42.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:41.906+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:42.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:41.906+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:42.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:41.906+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:42.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:42.529+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:42.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:42.529+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:42.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:42.529+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:42.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:42.529+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:42.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:42.582+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:42.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:42.582+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:42.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:42.582+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:42.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:42.583+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:43.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:42.956+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:43.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:42.956+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:43.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:42.956+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:43.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:42.957+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:43.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:43.590+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:43.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:43.590+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:43.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:43.590+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:43.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:43.592+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:43.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:43.524+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:43.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:43.524+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:43.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:43.524+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:43.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:43.524+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:44.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:43.962+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:44.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:43.962+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:44.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:43.962+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:44.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:43.962+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:44.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:44.567+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:44.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:44.567+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:44.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:44.567+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:44.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:44.568+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:44.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:44.528+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:44.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:44.528+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:44.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:44.528+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:44.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:44.529+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:45.415 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:45.000+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:45.416 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:45.000+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:45.416 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:45.000+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:45.416 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:45.000+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:45.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:45.503+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:45.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:45.503+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:45.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:45.503+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:45.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:45.504+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:45.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:45.550+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:45.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:45.550+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:45.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:45.550+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:45.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:45.551+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:46.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:46.006+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:46.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:46.006+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:46.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:46.006+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:46.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:46.007+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:46.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:07:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:07:46.454+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 5529 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:07:46.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:46.575+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:46.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:46.575+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:46.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:46.575+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:46.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:46.576+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:46.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:46.517+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:46.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:46.517+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:46.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:46.517+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:46.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:46.518+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:47.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:46.973+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:47.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:46.973+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:47.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:46.973+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:47.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:46.975+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:47.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:47.613+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:47.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:47.613+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:47.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:47.613+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:47.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:47.613+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:47.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:47.543+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:47.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:47.543+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:47.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:47.543+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:47.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:47.544+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:48.303 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:47.953+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:48.303 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:47.953+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:48.303 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:47.953+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:48.303 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:47.953+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:48.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:48.655+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:48.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:48.655+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:48.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:48.655+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:48.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:48.656+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:48.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:48.512+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:48.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:48.512+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:48.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:48.512+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:48.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:48.513+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:49.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:48.954+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:49.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:48.954+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:49.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:48.954+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:49.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:48.955+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:49.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:49.628+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:49.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:49.628+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:49.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:49.628+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:49.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:49.630+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:49.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:49.525+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:49.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:49.525+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:49.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:49.525+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:49.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:49.526+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:50.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:49.987+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:50.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:49.987+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:50.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:49.987+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:50.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:49.988+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:50.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:50.538+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:50.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:50.538+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:50.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:50.538+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:50.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:50.539+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:50.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:50.591+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:50.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:50.591+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:50.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:50.591+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:50.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:50.592+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:51.375 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:50.944+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:51.375 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:50.944+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:51.375 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:50.944+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:51.375 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:50.946+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:51.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:07:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:07:51.454+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 5623 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:07:51.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:51.515+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:51.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:51.515+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:51.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:51.515+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:51.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:51.516+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:51.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:51.551+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:51.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:51.551+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:51.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:51.551+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:51.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:51.551+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:52.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:51.916+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:52.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:51.916+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:52.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:51.916+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:52.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:51.916+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:52.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:52.547+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:52.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:52.547+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:52.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:52.547+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:52.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:52.548+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:52.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:52.516+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:52.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:52.516+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:52.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:52.516+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:52.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:52.517+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:53.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:52.918+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:53.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:52.918+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:53.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:52.918+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:53.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:52.919+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:53.905 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:53.510+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:53.905 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:53.510+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:53.905 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:53.510+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:53.906 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:53.511+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:53.906 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:53.534+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:53.906 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:53.534+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:53.906 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:53.534+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:53.906 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:53.535+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:54.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:53.900+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:54.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:53.900+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:54.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:53.900+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:54.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:53.901+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:54.886 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:54.498+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:54.887 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:54.498+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:54.887 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:54.498+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:54.887 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:54.498+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:54.887 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:54.576+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:54.887 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:54.576+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:54.887 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:54.576+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:54.887 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:54.577+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:55.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:54.881+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:55.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:54.881+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:55.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:54.881+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:55.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:54.883+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:55.855 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:55.506+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:55.855 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:55.506+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:55.855 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:55.506+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:55.855 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:55.506+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:55.855 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:55.561+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:55.856 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:55.561+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:55.856 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:55.561+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:55.856 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:55.561+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:56.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:55.851+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:56.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:55.851+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:56.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:55.851+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:56.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:55.852+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:56.874 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:56.466+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:56.874 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:56.466+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:56.874 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:56.466+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:56.874 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:56.467+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:56.874 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:07:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:07:56.456+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 5716 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:07:56.874 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:56.561+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:56.875 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:56.561+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:56.875 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:56.561+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:56.875 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:56.562+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:57.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:56.869+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:57.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:56.869+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:57.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:56.869+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:57.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:56.870+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:57.852 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:57.597+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:57.852 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:57.597+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:57.852 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:57.597+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:57.852 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:57.598+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:57.853 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:57.423+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:57.853 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:57.423+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:57.853 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:57.423+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:57.853 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:57.424+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:58.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:57.848+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:58.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:57.848+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:58.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:57.848+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:58.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:57.848+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:58.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:58.388+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:58.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:58.388+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:58.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:58.388+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:58.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:58.389+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:58.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:58.571+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:58.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:58.571+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:58.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:58.571+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:58.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:58.572+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:59.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:58.837+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:59.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:58.837+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:59.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:58.837+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:07:59.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:58.838+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:07:59.853 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:59.424+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:59.853 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:59.424+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:59.853 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:59.424+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:07:59.853 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:07:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:07:59.425+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:07:59.853 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:59.612+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:59.853 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:59.612+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:59.854 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:59.612+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:07:59.854 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:07:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:07:59.613+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:00.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:59.849+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:00.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:59.849+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:00.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:59.849+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:00.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:07:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:07:59.849+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:00.889 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:00.436+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:00.889 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:00.436+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:00.889 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:00.436+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:00.889 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:00.437+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:00.889 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:00.625+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:00.889 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:00.625+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:00.889 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:00.625+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:00.889 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:00.626+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:01.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:00.884+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:01.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:00.884+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:01.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:00.884+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:01.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:00.885+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:01.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:08:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:08:01.457+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 5810 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:08:01.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:01.651+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:01.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:01.651+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:01.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:01.651+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:01.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:01.651+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:01.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:01.419+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:01.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:01.419+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:01.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:01.419+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:01.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:01.420+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:02.395 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:01.933+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:02.396 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:01.933+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:02.396 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:01.933+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:02.396 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:01.934+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:02.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:02.391+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:02.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:02.391+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:02.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:02.391+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:02.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:02.392+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:02.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:02.634+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:02.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:02.634+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:02.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:02.634+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:02.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:02.634+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:03.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:02.946+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:03.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:02.946+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:03.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:02.946+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:03.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:02.947+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:03.662 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:03.406+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:03.662 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:03.406+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:03.662 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:03.406+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:03.662 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:03.407+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:03.662 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:03.630+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:03.663 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:03.630+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:03.663 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:03.630+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:03.663 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:03.631+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:04.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:03.918+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:04.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:03.918+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:04.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:03.918+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:04.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:03.919+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:04.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:04.453+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:04.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:04.454+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:04.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:04.454+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:04.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:04.455+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:04.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:04.673+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:04.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:04.673+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:04.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:04.673+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:04.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:04.674+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:05.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:04.936+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:05.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:04.936+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:05.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:04.936+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:05.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:04.936+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:05.727 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:05.459+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:05.728 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:05.459+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:05.728 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:05.459+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:05.728 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:05.460+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:06.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:05.723+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:06.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:05.723+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:06.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:05.723+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:06.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:05.724+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:06.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:05.940+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:06.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:05.940+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:06.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:05.940+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:06.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:05.940+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:06.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:08:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:08:06.458+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 5902 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:08:06.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:06.684+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:06.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:06.684+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:06.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:06.684+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:06.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:06.685+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:06.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:06.435+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:06.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:06.435+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:06.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:06.435+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:06.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:06.436+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:07.395 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:06.947+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:07.395 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:06.947+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:07.395 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:06.947+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:07.395 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:06.948+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:07.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:07.390+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:07.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:07.390+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:07.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:07.390+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:07.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:07.392+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:08.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:07.732+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:08.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:07.732+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:08.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:07.732+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:08.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:07.733+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:08.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:07.909+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:08.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:07.909+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:08.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:07.909+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:08.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:07.909+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:08.695 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:08.410+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:08.695 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:08.410+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:08.696 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:08.410+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:08.696 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:08.411+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:09.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:08.926+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:09.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:08.926+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:09.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:08.926+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:09.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:08.927+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:09.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:08.688+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:09.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:08.688+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:09.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:08.688+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:09.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:08.689+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:09.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:09.453+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:09.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:09.453+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:09.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:09.453+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:09.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:09.454+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:09.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:09.656+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:09.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:09.656+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:09.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:09.656+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:09.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:09.656+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:10.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:09.973+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:10.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:09.973+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:10.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:09.973+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:10.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:09.974+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:10.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:10.484+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:10.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:10.484+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:10.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:10.484+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:10.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:10.485+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:10.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:10.684+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:10.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:10.684+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:10.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:10.684+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:10.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:10.685+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:11.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:10.975+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:11.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:10.975+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:11.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:10.975+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:11.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:10.976+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:11.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:08:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:08:11.459+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 5996 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:08:11.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:11.658+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:11.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:11.658+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:11.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:11.658+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:11.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:11.659+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:11.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:11.472+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:11.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:11.472+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:11.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:11.472+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:11.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:11.473+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:12.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:11.995+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:12.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:11.995+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:12.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:11.995+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:12.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:11.996+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:12.879 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:12.425+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:12.879 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:12.425+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:12.879 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:12.425+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:12.879 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:12.427+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:12.880 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:12.621+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:12.880 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:12.621+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:12.880 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:12.621+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:12.880 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:12.621+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:13.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:12.970+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:13.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:12.970+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:13.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:12.970+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:13.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:12.971+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:13.660 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:13.404+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:13.661 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:13.404+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:13.661 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:13.404+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:13.661 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:13.405+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:13.989 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:13.656+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:13.989 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:13.656+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:13.989 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:13.656+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:13.989 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:13.657+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:14.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:13.984+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:14.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:13.984+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:14.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:13.984+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:14.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:13.985+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:14.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:14.450+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:14.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:14.450+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:14.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:14.450+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:14.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:14.451+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:14.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:14.620+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:14.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:14.620+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:14.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:14.620+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:14.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:14.621+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:15.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:14.977+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:15.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:14.977+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:15.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:14.977+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:15.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:14.978+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:15.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:15.427+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:15.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:15.427+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:15.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:15.427+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:15.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:15.427+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:15.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:15.652+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:15.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:15.652+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:15.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:15.652+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:15.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:15.653+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:16.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:16.002+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:16.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:16.002+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:16.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:16.002+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:16.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:16.003+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:16.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:08:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:08:16.460+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 6089 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:08:16.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:16.437+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:16.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:16.437+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:16.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:16.437+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:16.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:16.438+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:16.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:16.643+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:16.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:16.643+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:16.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:16.643+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:16.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:16.645+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:17.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:17.030+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:17.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:17.030+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:17.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:17.030+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:17.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:17.030+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:17.688 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:17.435+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:17.688 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:17.435+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:17.688 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:17.435+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:17.688 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:17.436+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:18.070 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:17.684+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:18.070 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:17.684+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:18.070 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:17.684+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:18.070 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:17.685+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:18.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:18.065+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:18.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:18.065+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:18.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:18.065+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:18.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:18.067+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:18.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:18.447+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:18.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:18.447+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:18.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:18.447+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:18.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:18.448+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:18.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:18.673+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:18.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:18.673+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:18.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:18.673+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:18.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:18.674+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:19.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:19.084+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:19.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:19.084+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:19.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:19.084+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:19.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:19.085+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:19.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:19.718+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:19.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:19.718+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:19.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:19.718+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:19.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:19.719+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:19.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:19.494+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:19.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:19.494+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:19.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:19.494+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:19.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:19.495+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:20.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:20.055+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:20.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:20.055+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:20.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:20.055+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:20.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:20.056+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:20.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:20.678+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:20.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:20.678+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:20.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:20.678+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:20.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:20.679+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:20.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:20.453+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:20.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:20.453+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:20.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:20.453+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:20.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:20.453+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:21.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:21.086+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:21.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:21.086+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:21.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:21.086+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:21.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:21.087+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:21.727 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:08:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:08:21.461+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 6182 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:08:21.727 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:21.493+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:21.728 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:21.493+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:21.728 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:21.493+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:21.728 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:21.494+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:22.049 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:21.723+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:22.049 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:21.723+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:22.049 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:21.723+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:22.049 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:21.724+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:22.045+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:22.045+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:22.045+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:22.045+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:22.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:22.710+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:22.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:22.710+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:22.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:22.710+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:22.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:22.711+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:22.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:22.497+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:22.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:22.497+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:22.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:22.497+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:22.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:22.498+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:23.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:23.016+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:23.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:23.016+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:23.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:23.016+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:23.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:23.017+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:23.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:23.462+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:23.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:23.462+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:23.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:23.462+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:23.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:23.463+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:23.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:23.704+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:23.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:23.704+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:23.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:23.704+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:23.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:23.705+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:24.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:24.052+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:24.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:24.052+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:24.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:24.052+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:24.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:24.053+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:24.704 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:24.447+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:24.704 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:24.447+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:24.704 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:24.447+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:24.704 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:24.448+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:25.059 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:24.700+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:25.059 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:24.700+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:25.059 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:24.700+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:25.059 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:24.701+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:25.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:25.055+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:25.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:25.055+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:25.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:25.055+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:25.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:25.055+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:25.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:25.497+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:25.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:25.497+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:25.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:25.497+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:25.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:25.498+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:25.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:25.699+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:25.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:25.699+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:25.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:25.699+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:25.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:25.700+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:26.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:26.042+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:26.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:26.042+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:26.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:26.042+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:26.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:26.043+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:26.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:08:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:08:26.461+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 6275 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:08:26.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:26.695+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:26.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:26.695+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:26.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:26.695+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:26.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:26.696+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:26.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:26.515+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:26.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:26.515+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:26.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:26.515+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:26.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:26.515+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:27.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:27.044+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:27.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:27.045+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:27.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:27.045+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:27.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:27.045+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:27.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:27.659+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:27.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:27.659+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:27.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:27.659+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:27.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:27.660+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:27.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:27.546+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:27.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:27.546+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:27.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:27.546+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:27.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:27.547+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:28.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:28.072+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:28.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:28.072+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:28.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:28.072+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:28.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:28.073+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:28.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:28.675+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:28.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:28.675+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:28.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:28.675+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:28.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:28.676+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:28.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:28.585+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:28.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:28.585+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:28.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:28.585+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:28.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:28.586+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:29.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:29.037+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:29.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:29.037+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:29.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:29.037+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:29.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:29.038+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:29.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:29.570+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:29.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:29.570+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:29.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:29.570+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:29.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:29.571+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:29.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:29.700+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:29.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:29.700+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:29.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:29.700+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:29.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:29.701+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:30.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:30.084+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:30.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:30.084+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:30.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:30.084+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:30.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:30.085+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:30.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:30.707+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:30.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:30.707+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:30.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:30.707+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:30.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:30.708+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:30.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:30.581+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:30.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:30.581+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:30.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:30.581+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:30.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:30.581+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:31.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:31.054+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:31.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:31.054+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:31.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:31.054+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:31.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:31.055+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:31.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:08:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:08:31.462+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 6369 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:08:31.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:31.686+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:31.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:31.686+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:31.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:31.686+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:31.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:31.687+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:31.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:31.600+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:31.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:31.600+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:31.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:31.600+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:31.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:31.601+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:32.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:32.006+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:32.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:32.006+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:32.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:32.006+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:32.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:32.006+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:32.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:32.619+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:32.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:32.619+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:32.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:32.619+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:32.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:32.620+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:32.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:32.720+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:32.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:32.720+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:32.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:32.720+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:32.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:32.721+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:33.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:32.988+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:33.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:32.988+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:33.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:32.988+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:33.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:32.989+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:33.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:33.626+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:33.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:33.626+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:33.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:33.626+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:33.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:33.627+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:33.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:33.672+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:33.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:33.672+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:33.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:33.672+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:33.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:33.673+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:34.382 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:34.036+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:34.382 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:34.036+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:34.382 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:34.036+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:34.382 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:34.037+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:34.642 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:34.604+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:34.643 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:34.604+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:34.643 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:34.604+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:34.643 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:34.605+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:34.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:34.638+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:34.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:34.638+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:34.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:34.638+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:34.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:34.639+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:35.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:35.042+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:35.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:35.042+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:35.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:35.042+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:35.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:35.043+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:35.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:35.668+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:35.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:35.668+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:35.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:35.668+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:35.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:35.668+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:35.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:35.619+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:35.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:35.619+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:35.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:35.619+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:35.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:35.620+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:36.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:36.015+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:36.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:36.015+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:36.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:36.015+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:36.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:36.016+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:36.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:08:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:08:36.463+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 6462 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:08:36.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:36.626+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:36.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:36.626+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:36.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:36.626+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:36.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:36.627+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:36.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:36.619+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:36.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:36.619+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:36.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:36.619+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:36.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:36.619+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:37.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:37.049+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:37.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:37.049+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:37.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:37.049+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:37.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:37.050+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:37.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:37.636+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:37.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:37.636+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:37.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:37.636+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:37.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:37.637+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:37.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:37.656+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:37.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:37.656+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:37.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:37.656+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:37.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:37.657+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:38.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:38.053+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:38.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:38.053+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:38.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:38.053+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:38.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:38.053+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:38.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:38.590+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:38.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:38.590+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:38.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:38.590+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:38.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:38.590+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:38.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:38.618+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:38.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:38.618+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:38.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:38.618+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:38.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:38.619+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:39.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:39.030+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:39.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:39.030+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:39.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:39.030+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:39.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:39.031+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:39.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T22:08:39.663+0000 7f128a252700 0 monclient(hunting): authenticate timed out after 300 2024-02-09T22:08:39.668 INFO:teuthology.orchestra.run.smithi006.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-02-09T22:08:39.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:39.575+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:39.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:39.575+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:39.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:39.575+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:39.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:39.575+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:39.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:39.547+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:39.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:39.547+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:39.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:39.547+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:39.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:39.548+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:40.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:40.038+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:40.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:40.038+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:40.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:40.038+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:40.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:40.039+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:40.565 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-02-09T22:08:40.841 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/mon.smithi006/config 2024-02-09T22:08:40.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:40.549+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:40.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:40.549+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:40.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:40.549+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:40.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:40.550+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:40.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:40.509+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:40.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:40.509+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:40.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:40.509+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:40.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:40.510+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:41.240 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:41.026+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:41.240 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:41.026+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:41.240 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:41.026+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:41.240 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:41.027+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:41.493 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:08:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:08:41.464+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 6555 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:08:41.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:41.565+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:41.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:41.565+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:41.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:41.565+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:41.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:41.566+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:41.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:41.509+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:41.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:41.509+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:41.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:41.509+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:41.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:41.510+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:42.347 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:42.026+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:42.347 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:42.026+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:42.347 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:42.026+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:42.348 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:42.027+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:42.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:42.563+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:42.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:42.563+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:42.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:42.563+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:42.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:42.564+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:42.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:42.534+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:42.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:42.534+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:42.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:42.534+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:42.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:42.534+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:43.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:43.020+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:43.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:43.020+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:43.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:43.020+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:43.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:43.021+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:43.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:43.592+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:43.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:43.592+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:43.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:43.592+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:43.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:43.593+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:43.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:43.566+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:43.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:43.566+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:43.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:43.566+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:43.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:43.567+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:44.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:44.037+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:44.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:44.037+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:44.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:44.037+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:44.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:44.037+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:44.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:44.544+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:44.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:44.544+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:44.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:44.544+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:44.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:44.545+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:44.911 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:44.536+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:44.911 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:44.536+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:44.911 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:44.536+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:44.911 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:44.537+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:45.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:45.068+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:45.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:45.068+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:45.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:45.068+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:45.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:45.069+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:45.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:45.506+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:45.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:45.506+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:45.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:45.506+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:45.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:45.506+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:45.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:45.556+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:45.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:45.556+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:45.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:45.556+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:45.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:45.557+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:46.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:46.059+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:46.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:46.059+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:46.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:46.059+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:46.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:46.059+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:46.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:08:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:08:46.465+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 6648 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:08:46.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:46.527+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:46.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:46.527+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:46.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:46.527+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:46.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:46.527+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:46.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:46.586+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:46.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:46.586+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:46.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:46.586+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:46.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:46.587+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:47.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:47.105+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.890278+0000 front 2024-02-09T21:58:19.890132+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:47.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:47.105+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.890082+0000 front 2024-02-09T21:58:19.890191+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:47.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:47.105+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.890216+0000 front 2024-02-09T21:58:19.890305+0000 (oldest deadline 2024-02-09T21:58:45.790140+0000) 2024-02-09T22:08:47.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:47.106+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:47.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:47.504+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:19.444243+0000 front 2024-02-09T21:58:19.444492+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:47.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:47.504+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:19.444560+0000 front 2024-02-09T21:58:19.444613+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:47.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:47.504+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:19.444530+0000 front 2024-02-09T21:58:19.444577+0000 (oldest deadline 2024-02-09T21:58:44.144309+0000) 2024-02-09T22:08:47.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:47.505+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:47.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:47.620+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:47.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:47.620+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:47.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:47.620+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:47.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:47.621+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:48.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:48.088+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:48.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:48.465+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:48.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:48.633+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:48.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:48.633+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:48.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:48.633+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:48.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:48.633+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:49.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:49.139+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:49.742 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:49.451+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:49.743 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:49.642+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:49.743 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:49.642+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:49.743 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:49.642+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:49.743 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:49.643+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:50.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:50.435+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:50.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:50.189+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:51.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:50.685+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:51.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:50.685+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:51.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:50.685+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:51.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:50.686+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:51.481 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:08:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:08:51.466+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 6743 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:08:51.482 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:51.456+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:51.482 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:51.216+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:51.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:51.690+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:51.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:51.690+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:51.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:51.690+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:51.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:51.690+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:52.481 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:52.180+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:52.814 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:52.477+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:52.814 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:52.658+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:52.814 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:52.658+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:52.815 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:52.658+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:52.815 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:52.659+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:53.446 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:53.158+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:53.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:53.442+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:53.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:53.664+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:53.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:53.664+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:53.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:53.664+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:53.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:53.665+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:54.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:54.151+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:54.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:54.411+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:54.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:54.660+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:54.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:54.660+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:54.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:54.660+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:54.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:54.661+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:55.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:55.149+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:55.886 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:55.429+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:55.886 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:55.643+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:55.887 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:55.643+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:55.887 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:55.643+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:55.887 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:55.644+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:56.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:56.125+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:56.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:08:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:08:56.467+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 6837 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:08:56.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:56.470+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:56.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:56.684+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:56.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:56.684+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:56.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:56.684+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:56.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:56.685+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:57.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:57.151+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:57.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:57.505+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:57.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:57.647+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:57.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:57.647+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:57.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:57.647+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:57.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:57.648+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:58.451 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:58.179+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:58.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:58.518+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:58.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:58.688+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:58.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:58.688+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:58.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:58.688+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:58.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:58.689+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:59.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:08:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:08:59.146+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:08:59.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:08:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:08:59.518+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:08:59.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:59.672+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:59.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:59.672+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:59.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:59.672+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:08:59.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:08:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:08:59.673+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:00.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:00.139+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:00.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:00.504+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:00.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:00.691+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:09:00.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:00.691+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:09:00.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:00.691+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:09:00.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:00.692+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:01.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:01.096+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:01.733 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:09:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:09:01.468+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 6931 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:09:01.734 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:01.493+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:02.030 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:01.729+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 since back 2024-02-09T21:58:22.218236+0000 front 2024-02-09T21:58:22.218305+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:09:02.031 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:01.729+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 since back 2024-02-09T21:58:22.218251+0000 front 2024-02-09T21:58:22.218324+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:09:02.031 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:01.729+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 since back 2024-02-09T21:58:22.218391+0000 front 2024-02-09T21:58:22.218355+0000 (oldest deadline 2024-02-09T21:58:45.118325+0000) 2024-02-09T22:09:02.031 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:01.730+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:02.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:02.112+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:02.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:02.523+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:02.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:02.738+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:03.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:03.120+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:03.751 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:03.569+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:04.132 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:03.747+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:04.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:04.128+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:04.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:04.619+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:04.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:04.722+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:05.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:05.117+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:05.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:05.651+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:05.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:05.736+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:06.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:06.073+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:06.754 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:09:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:09:06.469+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 7024 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:09:06.755 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:06.652+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:07.082 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:06.750+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:07.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:07.078+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:08.062 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:07.654+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:08.062 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:07.745+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:08.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:08.058+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:09.056 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:08.698+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:09.057 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:08.765+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:09.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:09.052+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:09.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:09.052+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:09.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:09.052+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:09.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:09.053+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:10.107 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:09.731+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:10.107 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:09.810+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:10.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:10.103+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:10.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:10.103+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:10.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:10.103+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:10.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:10.103+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:11.073 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:10.684+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:11.074 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:10.775+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:11.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:11.069+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:11.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:11.069+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:11.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:11.069+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:11.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:11.070+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:11.738 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:09:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:09:11.470+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 7119 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:09:11.739 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:11.693+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:11.739 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:11.693+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:11.739 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:11.693+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:11.739 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:11.694+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:12.063 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:11.734+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:12.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:12.059+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:12.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:12.059+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:12.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:12.059+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:12.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:12.060+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:13.017 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:12.762+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:13.017 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:12.681+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:13.017 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:12.681+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:13.017 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:12.681+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:13.017 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:12.682+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:13.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:13.012+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:13.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:13.012+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:13.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:13.012+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:13.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:13.014+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:13.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:13.710+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:13.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:13.710+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:13.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:13.710+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:13.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:13.711+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:13.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:13.731+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:14.318 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:13.964+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:14.318 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:13.964+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:14.318 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:13.964+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:14.318 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:13.965+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:15.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:14.705+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:15.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:14.698+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:15.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:14.698+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:15.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:14.698+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:15.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:14.699+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:15.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:14.931+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:15.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:14.931+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:15.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:14.931+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:15.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:14.932+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:15.950 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:15.698+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:15.950 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:15.698+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:15.950 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:15.698+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:15.950 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:15.699+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:15.950 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:15.686+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:16.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:15.945+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:16.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:15.945+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:16.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:15.945+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:16.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:15.946+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:16.733 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:09:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:09:16.471+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 7213 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:09:16.733 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:16.666+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:17.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:16.728+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:17.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:16.728+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:17.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:16.728+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:17.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:16.730+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:17.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:16.966+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:17.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:16.966+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:17.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:16.966+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:17.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:16.968+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:17.983 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:17.690+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:17.984 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:17.690+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:17.984 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:17.690+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:17.984 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:17.691+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:17.984 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:17.704+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:18.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:17.979+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:18.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:17.979+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:18.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:17.979+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:18.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:17.980+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:19.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:18.726+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:19.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:18.726+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:19.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:18.726+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:19.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:18.727+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:19.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:18.946+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:19.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:18.946+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:19.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:18.946+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:19.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:18.946+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:19.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:18.719+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:20.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:19.735+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:20.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:19.735+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:20.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:19.735+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:20.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:19.735+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:20.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:19.696+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:20.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:19.914+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:20.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:19.914+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:20.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:19.914+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:20.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:19.914+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:21.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:20.715+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:21.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:20.718+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:21.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:20.718+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:21.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:20.718+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:21.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:20.720+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:21.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:20.925+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:21.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:20.925+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:21.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:20.925+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:21.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:20.925+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:21.749 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:09:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:09:21.472+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 7308 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:09:22.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:21.745+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:22.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:21.745+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:22.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:21.745+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:22.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:21.746+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:22.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:21.894+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:22.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:21.894+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:22.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:21.894+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:22.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:21.894+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:22.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:21.754+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:23.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:22.700+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:23.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:22.700+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:23.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:22.700+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:23.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:22.701+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:23.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:22.878+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:23.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:22.878+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:23.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:22.878+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:23.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:22.878+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:23.160 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:22.756+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:23.763 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:23.667+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:23.763 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:23.667+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:23.763 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:23.667+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:23.763 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:23.668+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:24.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:23.862+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:24.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:23.862+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:24.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:23.862+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:24.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:23.863+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:24.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:23.759+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:24.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:23.759+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:24.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:23.759+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:24.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:23.760+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:24.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:24.640+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:24.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:24.640+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:24.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:24.640+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:24.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:24.640+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:24.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:24.854+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:24.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:24.854+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:24.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:24.854+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:24.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:24.855+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:24.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:24.724+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:24.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:24.724+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:24.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:24.724+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:24.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:24.724+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:26.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:25.657+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:26.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:25.657+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:26.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:25.657+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:26.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:25.658+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:26.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:25.819+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:26.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:25.819+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:26.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:25.819+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:26.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:25.819+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:26.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:25.748+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:26.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:25.748+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:26.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:25.748+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:26.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:25.749+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:26.747 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:09:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:09:26.472+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 7401 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:09:26.748 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:26.691+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:26.748 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:26.691+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:26.748 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:26.691+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:26.748 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:26.692+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:27.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:26.788+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:27.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:26.788+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:27.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:26.788+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:27.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:26.789+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:27.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:26.742+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:27.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:26.742+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:27.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:26.742+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:27.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:26.743+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:28.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:27.659+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:28.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:27.659+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:28.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:27.659+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:28.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:27.660+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:28.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:27.767+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:28.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:27.767+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:28.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:27.767+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:28.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:27.767+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:28.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:27.773+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:28.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:27.773+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:28.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:27.773+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:28.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:27.773+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:29.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:28.699+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:29.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:28.699+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:29.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:28.699+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:29.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:28.699+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:29.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:28.751+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:29.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:28.751+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:29.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:28.751+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:29.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:28.751+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:29.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:28.798+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:29.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:28.798+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:29.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:28.798+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:29.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:28.799+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:30.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:29.694+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:30.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:29.694+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:30.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:29.694+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:30.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:29.694+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:30.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:29.765+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:30.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:29.765+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:30.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:29.765+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:30.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:29.765+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:30.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:29.846+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:30.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:29.846+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:30.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:29.846+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:30.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:29.847+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:31.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:30.677+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:31.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:30.677+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:31.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:30.677+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:31.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:30.677+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:31.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:30.730+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:31.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:30.730+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:31.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:30.730+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:31.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:30.731+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:31.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:30.803+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:31.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:30.803+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:31.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:30.803+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:31.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:30.803+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:31.786 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:09:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:09:31.473+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 7496 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:09:31.786 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:31.640+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:31.786 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:31.640+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:31.786 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:31.640+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:31.787 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:31.640+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:31.787 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:31.709+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:31.787 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:31.709+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:31.787 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:31.709+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:31.787 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:31.710+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:32.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:31.781+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:32.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:31.781+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:32.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:31.781+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:32.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:31.782+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:33.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:32.654+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:33.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:32.654+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:33.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:32.654+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:33.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:32.655+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:33.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:32.689+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:33.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:32.689+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:33.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:32.689+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:33.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:32.690+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:33.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:32.782+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:33.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:32.782+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:33.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:32.782+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:33.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:32.783+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:33.804 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:33.691+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:33.804 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:33.691+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:33.804 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:33.691+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:33.805 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:33.693+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:33.805 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:33.661+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:33.805 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:33.661+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:33.805 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:33.661+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:33.805 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:33.661+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:34.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:33.800+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:34.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:33.800+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:34.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:33.800+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:34.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:33.801+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:34.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:34.808+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:34.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:34.808+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:34.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:34.808+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:34.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:34.809+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:34.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:34.672+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:34.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:34.672+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:34.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:34.672+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:34.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:34.673+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:34.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:34.634+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:34.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:34.634+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:34.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:34.634+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:34.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:34.635+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:35.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:35.641+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:35.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:35.641+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:35.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:35.641+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:35.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:35.643+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:35.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:35.608+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:35.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:35.608+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:35.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:35.608+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:35.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:35.609+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:35.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:35.813+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:35.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:35.813+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:35.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:35.813+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:35.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:35.813+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:36.831 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:09:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:09:36.475+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 7590 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:09:36.831 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:36.617+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:36.831 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:36.617+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:36.831 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:36.617+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:36.831 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:36.619+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:36.832 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:36.626+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:36.832 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:36.626+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:36.832 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:36.626+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:36.832 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:36.627+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:37.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:36.826+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:37.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:36.826+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:37.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:36.826+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:37.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:36.827+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:38.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:37.664+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:38.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:37.664+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:38.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:37.664+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:38.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:37.665+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:38.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:37.654+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:38.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:37.654+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:38.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:37.654+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:38.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:37.655+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:38.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:37.831+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:38.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:37.831+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:38.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:37.831+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:38.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:37.831+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:38.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:38.691+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:38.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:38.691+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:38.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:38.691+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:38.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:38.693+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:38.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:38.605+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:38.910 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:38.605+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:38.910 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:38.605+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:38.910 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:38.606+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:38.910 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:38.795+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:38.911 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:38.795+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:38.911 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:38.795+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:38.911 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:38.796+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:39.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:39.582+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:39.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:39.582+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:39.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:39.582+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:39.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:39.583+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:39.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:39.812+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:39.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:39.812+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:39.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:39.812+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:39.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:39.813+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:39.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:39.739+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:39.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:39.739+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:39.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:39.739+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:39.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:39.740+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:40.807 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:40.553+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:40.807 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:40.553+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:40.807 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:40.553+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:40.807 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:40.553+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:40.807 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:40.802+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:40.808 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:40.802+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:40.808 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:40.802+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:40.808 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:40.803+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:40.808 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:40.738+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:40.808 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:40.738+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:40.808 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:40.738+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:40.808 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:40.738+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:41.774 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:09:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:09:41.477+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 7684 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:09:41.774 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:41.711+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:41.774 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:41.711+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:41.774 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:41.711+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:41.774 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:41.712+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:41.775 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:41.535+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:41.775 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:41.535+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:41.775 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:41.535+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:41.775 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:41.536+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:42.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:41.770+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:42.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:41.770+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:42.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:41.770+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:42.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:41.771+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:42.757 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:42.502+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:42.757 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:42.503+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:42.758 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:42.503+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:42.758 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:42.503+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:42.758 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:42.752+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:42.758 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:42.752+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:42.758 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:42.752+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:42.758 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:42.753+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:42.758 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:42.742+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:42.758 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:42.742+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:42.758 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:42.742+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:42.759 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:42.742+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:43.777 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:43.720+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:43.777 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:43.720+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:43.777 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:43.720+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:43.777 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:43.721+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:43.777 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:43.547+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:43.777 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:43.547+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:43.777 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:43.547+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:43.777 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:43.548+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:44.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:43.772+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:44.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:43.772+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:44.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:43.772+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:44.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:43.773+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:44.809 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:44.734+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:44.809 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:44.734+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:44.809 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:44.734+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:44.809 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:44.735+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:44.809 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:44.535+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:44.809 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:44.535+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:44.809 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:44.535+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:44.809 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:44.537+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:45.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:44.805+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:45.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:44.805+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:45.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:44.805+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:45.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:44.805+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:45.854 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:45.722+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:45.854 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:45.722+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:45.854 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:45.722+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:45.854 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:45.723+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:45.854 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:45.508+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:45.854 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:45.508+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:45.855 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:45.508+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:45.855 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:45.509+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:46.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:45.850+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:46.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:45.850+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:46.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:45.850+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:46.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:45.851+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:46.754 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:09:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:09:46.478+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 7777 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:09:46.754 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:46.508+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:46.754 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:46.508+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:46.754 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:46.508+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:46.754 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:46.509+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:47.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:46.750+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:47.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:46.750+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:47.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:46.750+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:47.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:46.751+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:47.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:46.839+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:47.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:46.839+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:47.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:46.839+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:47.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:46.839+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:47.739 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:47.474+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:47.739 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:47.474+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:47.740 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:47.474+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:47.740 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:47.475+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:48.111 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:47.735+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:48.111 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:47.735+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:48.111 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:47.735+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:48.111 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:47.736+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:48.111 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:47.819+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:48.111 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:47.819+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:48.111 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:47.819+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:48.111 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:47.820+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:48.871 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:48.724+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:48.871 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:48.724+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:48.871 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:48.724+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:48.871 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:48.725+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:48.872 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:48.525+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:48.872 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:48.525+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:48.872 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:48.525+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:48.872 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:48.526+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:49.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:48.866+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:49.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:48.866+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:49.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:48.866+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:49.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:48.867+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:49.773 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:49.507+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:49.773 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:49.507+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:49.773 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:49.507+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:49.773 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:49.508+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:50.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:49.769+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:50.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:49.769+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:50.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:49.769+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:50.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:49.770+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:50.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:49.838+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:50.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:49.838+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:50.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:49.838+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:50.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:49.839+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:50.826 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:50.726+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:50.826 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:50.726+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:50.826 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:50.726+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:50.826 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:50.726+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:50.827 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:50.555+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:50.827 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:50.555+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:50.827 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:50.555+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:50.827 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:50.556+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:51.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:50.822+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:51.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:50.822+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:51.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:50.822+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:51.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:50.823+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:51.778 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:09:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:09:51.479+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 7872 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:09:51.778 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:51.533+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:51.778 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:51.533+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:51.778 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:51.533+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:51.778 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:51.534+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:52.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:51.774+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:52.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:51.774+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:52.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:51.774+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:52.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:51.775+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:52.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:51.809+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:52.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:51.809+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:52.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:51.809+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:52.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:51.810+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:52.808 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:52.552+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:52.809 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:52.552+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:52.809 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:52.552+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:52.809 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:52.553+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:53.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:52.824+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:53.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:52.824+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:53.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:52.824+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:53.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:52.825+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:53.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:52.804+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:53.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:52.804+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:53.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:52.804+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:53.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:52.805+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:53.818 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:53.555+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:53.818 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:53.555+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:53.818 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:53.555+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:53.818 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:53.556+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:54.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:53.819+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:54.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:53.819+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:54.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:53.819+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:54.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:53.820+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:54.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:53.814+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:54.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:53.814+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:54.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:53.814+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:54.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:53.815+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:54.776 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:54.509+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:54.776 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:54.509+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:54.776 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:54.509+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:54.777 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:54.510+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:55.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:54.772+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:55.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:54.772+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:55.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:54.772+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:55.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:54.773+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:55.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:54.842+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:55.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:54.842+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:55.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:54.842+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:55.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:54.843+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:55.787 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:55.517+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:55.787 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:55.517+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:55.788 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:55.517+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:55.788 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:55.517+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:56.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:55.795+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:56.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:55.795+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:56.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:55.795+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:56.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:55.795+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:56.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:55.783+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:56.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:55.783+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:56.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:55.783+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:56.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:55.784+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:56.777 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:09:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:09:56.480+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 7966 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:09:56.777 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:56.497+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:56.778 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:56.497+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:56.778 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:56.497+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:56.778 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:56.498+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:57.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:56.773+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:57.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:56.773+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:57.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:56.773+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:57.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:56.774+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:57.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:56.791+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:57.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:56.791+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:57.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:56.791+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:57.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:56.792+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:57.810 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:57.755+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:57.810 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:57.755+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:57.811 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:57.755+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:57.811 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:57.755+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:57.811 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:57.514+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:57.811 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:57.514+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:57.811 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:57.514+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:57.811 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:57.515+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:58.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:57.806+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:58.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:57.806+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:58.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:57.806+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:58.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:57.807+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:58.787 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:58.486+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:58.787 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:58.486+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:58.787 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:58.486+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:58.787 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:58.486+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:59.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:58.808+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:59.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:58.808+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:59.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:58.808+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:09:59.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:58.808+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:09:59.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:58.783+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:59.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:58.783+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:59.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:58.783+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:09:59.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:58.783+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:09:59.772 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:59.485+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:59.772 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:59.485+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:59.772 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:59.485+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:09:59.772 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:09:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:09:59.486+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:00.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:59.857+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:00.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:59.857+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:00.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:59.857+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:00.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:09:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:09:59.858+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:00.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:59.768+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:00.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:59.768+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:00.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:59.768+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:00.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:09:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:09:59.768+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:00.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:00.498+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:00.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:00.498+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:00.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:00.498+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:00.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:00.499+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:01.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:00.857+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:01.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:00.857+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:01.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:00.857+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:01.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:00.858+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:01.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:00.790+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:01.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:00.790+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:01.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:00.790+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:01.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:00.791+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:01.828 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:10:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:10:01.481+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 8059 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:10:01.829 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:01.472+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:01.829 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:01.472+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:01.829 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:01.472+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:01.829 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:01.473+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:02.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:01.876+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:02.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:01.876+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:02.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:01.876+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:02.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:01.877+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:02.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:01.824+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:02.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:01.824+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:02.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:01.824+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:02.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:01.825+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:02.819 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:02.490+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:02.819 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:02.490+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:02.819 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:02.490+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:02.819 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:02.491+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:03.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:02.872+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:03.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:02.872+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:03.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:02.872+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:03.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:02.873+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:03.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:02.814+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:03.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:02.814+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:03.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:02.814+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:03.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:02.815+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:03.844 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:03.505+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:03.844 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:03.505+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:03.844 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:03.505+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:03.844 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:03.506+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:04.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:03.916+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:04.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:03.916+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:04.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:03.916+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:04.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:03.917+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:04.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:03.840+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:04.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:03.840+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:04.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:03.840+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:04.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:03.841+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:04.868 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:04.500+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:04.868 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:04.500+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:04.868 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:04.500+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:04.868 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:04.500+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:05.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:04.938+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:05.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:04.938+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:05.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:04.938+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:05.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:04.938+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:05.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:04.864+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:05.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:04.864+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:05.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:04.864+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:05.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:04.865+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:05.836 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:05.519+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:05.836 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:05.519+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:05.836 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:05.519+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:05.837 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:05.519+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:06.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:05.924+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:06.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:05.924+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:06.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:05.924+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:06.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:05.924+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:06.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:05.832+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:06.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:05.832+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:06.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:05.832+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:06.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:05.833+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:06.874 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:10:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:10:06.482+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 8150 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:10:06.874 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:06.552+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:06.874 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:06.552+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:06.874 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:06.552+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:06.874 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:06.553+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:07.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:06.966+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:07.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:06.966+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:07.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:06.966+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:07.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:06.967+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:07.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:06.870+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:07.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:06.870+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:07.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:06.870+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:07.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:06.871+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:07.827 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:07.567+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:07.827 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:07.567+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:07.827 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:07.567+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:07.827 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:07.568+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:08.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:07.938+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:08.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:07.938+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:08.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:07.938+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:08.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:07.939+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:08.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:07.823+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:08.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:07.823+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:08.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:07.823+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:08.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:07.824+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:08.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:08.614+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:08.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:08.614+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:08.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:08.614+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:08.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:08.615+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:08.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:08.830+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:08.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:08.830+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:08.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:08.830+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:08.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:08.831+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:09.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:08.909+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:09.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:08.909+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:09.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:08.909+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:09.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:08.910+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:09.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:09.644+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:09.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:09.644+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:09.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:09.644+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:09.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:09.645+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:09.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:09.785+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:09.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:09.785+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:09.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:09.785+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:09.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:09.785+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:10.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:09.922+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:10.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:09.922+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:10.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:09.922+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:10.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:09.923+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:10.880 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:10.779+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:10.880 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:10.779+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:10.880 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:10.779+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:10.880 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:10.780+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:10.881 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:10.615+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:10.881 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:10.615+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:10.881 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:10.615+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:10.881 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:10.617+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:11.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:10.876+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:11.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:10.876+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:11.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:10.876+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:11.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:10.877+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:11.809 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:10:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:10:11.483+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 8245 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:10:11.809 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:11.606+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:11.809 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:11.606+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:11.809 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:11.606+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:11.810 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:11.607+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:12.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:11.875+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:12.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:11.875+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:12.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:11.875+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:12.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:11.875+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:12.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:11.805+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:12.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:11.805+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:12.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:11.805+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:12.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:11.806+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:12.853 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:12.593+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:12.853 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:12.593+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:12.853 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:12.593+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:12.853 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:12.593+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:13.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:12.879+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:13.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:12.879+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:13.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:12.879+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:13.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:12.879+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:13.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:12.848+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:13.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:12.848+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:13.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:12.848+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:13.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:12.849+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:13.848 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:13.639+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:13.848 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:13.639+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:13.848 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:13.639+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:13.848 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:13.640+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:14.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:13.843+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:14.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:13.843+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:14.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:13.843+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:14.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:13.845+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:14.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:13.845+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:14.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:13.845+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:14.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:13.845+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:14.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:13.847+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:15.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:14.865+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:15.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:14.865+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:15.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:14.865+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:15.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:14.866+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:15.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:14.885+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:15.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:14.885+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:15.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:14.885+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:15.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:14.886+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:15.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:14.662+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:15.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:14.662+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:15.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:14.662+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:15.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:14.663+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:16.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:15.849+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:16.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:15.849+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:16.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:15.849+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:16.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:15.850+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:16.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:15.936+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:16.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:15.936+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:16.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:15.936+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:16.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:15.937+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:16.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:15.698+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:16.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:15.698+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:16.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:15.698+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:16.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:15.699+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:16.875 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:10:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:10:16.484+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 8338 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:10:16.875 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:16.685+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:16.876 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:16.685+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:16.876 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:16.685+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:16.876 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:16.686+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:17.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:16.921+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:17.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:16.921+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:17.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:16.921+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:17.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:16.922+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:17.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:16.871+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:17.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:16.871+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:17.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:16.871+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:17.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:16.872+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:18.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:17.736+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:18.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:17.736+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:18.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:17.736+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:18.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:17.737+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:18.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:17.885+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:18.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:17.885+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:18.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:17.885+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:18.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:17.885+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:18.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:17.874+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:18.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:17.874+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:18.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:17.874+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:18.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:17.875+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:19.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:18.701+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:19.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:18.701+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:19.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:18.701+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:19.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:18.702+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:19.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:18.871+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:19.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:18.871+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:19.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:18.871+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:19.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:18.872+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:19.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:18.912+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:19.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:18.912+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:19.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:18.912+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:19.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:18.913+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:20.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:19.848+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:20.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:19.848+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:20.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:19.848+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:20.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:19.849+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:20.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:19.671+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:20.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:19.671+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:20.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:19.671+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:20.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:19.672+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:20.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:19.893+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:20.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:19.893+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:20.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:19.893+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:20.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:19.893+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:21.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:20.837+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:21.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:20.837+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:21.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:20.837+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:21.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:20.838+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:21.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:20.661+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:21.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:20.661+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:21.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:20.661+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:21.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:20.662+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:21.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:20.869+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:21.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:20.869+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:21.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:20.869+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:21.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:20.870+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:21.794 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:10:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:10:21.485+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 8432 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:10:21.794 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:21.648+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:21.794 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:21.648+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:21.795 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:21.648+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:21.795 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:21.649+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:22.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:21.790+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:22.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:21.790+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:22.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:21.790+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:22.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:21.791+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:22.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:21.877+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:22.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:21.877+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:22.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:21.877+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:22.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:21.878+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:23.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:22.693+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:23.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:22.693+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:23.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:22.693+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:23.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:22.694+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:23.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:22.817+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:23.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:22.817+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:23.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:22.817+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:23.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:22.817+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:23.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:22.883+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:23.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:22.883+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:23.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:22.883+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:23.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:22.885+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:23.836 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:23.658+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:23.836 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:23.658+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:23.836 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:23.658+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:23.836 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:23.659+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:24.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:23.840+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:24.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:23.840+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:24.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:23.840+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:24.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:23.841+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:24.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:23.832+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:24.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:23.832+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:24.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:23.832+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:24.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:23.833+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:24.974 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:24.864+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:24.974 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:24.864+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:24.974 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:24.864+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:24.974 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:24.865+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:24.975 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:24.883+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:24.975 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:24.883+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:24.975 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:24.883+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:24.975 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:24.884+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:24.975 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:24.656+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:24.975 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:24.656+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:24.975 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:24.656+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:24.975 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:24.657+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:26.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:25.833+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:26.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:25.833+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:26.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:25.833+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:26.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:25.833+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:26.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:25.684+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:26.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:25.684+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:26.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:25.684+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:26.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:25.686+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:26.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:25.848+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:26.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:25.848+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:26.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:25.848+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:26.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:25.849+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:26.822 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:10:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:10:26.486+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 8524 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:10:26.822 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:26.721+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:26.822 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:26.721+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:26.822 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:26.721+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:26.822 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:26.721+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:27.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:26.875+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:27.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:26.875+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:27.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:26.875+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:27.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:26.876+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:27.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:26.818+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:27.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:26.818+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:27.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:26.818+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:27.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:26.819+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:28.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:27.704+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:28.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:27.704+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:28.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:27.704+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:28.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:27.705+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:28.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:27.870+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:28.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:27.870+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:28.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:27.870+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:28.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:27.871+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:28.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:27.828+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:28.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:27.828+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:28.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:27.828+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:28.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:27.829+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:29.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:28.782+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:29.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:28.782+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:29.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:28.782+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:29.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:28.783+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:29.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:28.736+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:29.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:28.736+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:29.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:28.736+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:29.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:28.737+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:29.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:28.848+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:29.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:28.848+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:29.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:28.848+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:29.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:28.849+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:30.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:29.803+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:30.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:29.803+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:30.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:29.803+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:30.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:29.804+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:30.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:29.695+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:30.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:29.695+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:30.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:29.695+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:30.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:29.696+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:30.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:29.808+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:30.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:29.808+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:30.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:29.808+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:30.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:29.809+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:31.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:30.826+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:31.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:30.826+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:31.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:30.826+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:31.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:30.827+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:31.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:30.677+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:31.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:30.677+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:31.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:30.677+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:31.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:30.677+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:31.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:30.810+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:31.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:30.810+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:31.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:30.810+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:31.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:30.810+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:31.788 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:10:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:10:31.487+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 8620 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:10:31.788 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:31.663+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:31.788 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:31.663+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:31.788 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:31.663+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:31.788 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:31.663+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:32.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:31.785+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:32.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:31.785+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:32.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:31.785+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:32.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:31.785+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:32.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:31.783+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:32.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:31.784+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:32.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:31.784+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:32.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:31.785+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:33.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:32.763+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:33.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:32.763+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:33.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:32.763+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:33.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:32.764+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:33.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:32.757+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:33.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:32.757+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:33.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:32.757+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:33.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:32.758+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:33.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:32.706+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:33.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:32.706+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:33.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:32.706+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:33.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:32.707+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:33.750 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:33.736+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:33.750 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:33.736+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:33.750 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:33.736+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:33.750 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:33.737+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:34.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:33.765+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:34.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:33.765+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:34.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:33.765+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:34.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:33.765+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:34.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:33.746+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:34.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:33.746+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:34.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:33.746+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:34.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:33.747+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:35.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:34.784+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:35.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:34.784+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:35.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:34.784+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:35.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:34.785+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:35.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:34.735+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:35.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:34.735+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:35.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:34.735+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:35.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:34.736+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:35.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:34.798+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:35.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:34.798+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:35.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:34.798+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:35.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:34.798+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:36.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:35.780+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:36.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:35.780+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:36.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:35.780+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:36.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:35.781+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:36.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:35.686+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:36.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:35.686+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:36.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:35.686+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:36.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:35.687+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:36.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:35.800+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:36.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:35.800+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:36.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:35.800+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:36.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:35.800+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:36.794 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:10:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:10:36.488+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 8713 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:10:36.794 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:36.667+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:36.794 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:36.667+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:36.794 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:36.667+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:36.794 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:36.668+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:37.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:36.790+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:37.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:36.790+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:37.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:36.790+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:37.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:36.792+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:37.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:36.790+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:37.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:36.790+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:37.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:36.790+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:37.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:36.791+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:38.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:37.841+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:38.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:37.841+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:38.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:37.841+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:38.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:37.842+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:38.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:37.751+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:38.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:37.751+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:38.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:37.751+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:38.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:37.752+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:38.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:37.705+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:38.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:37.705+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:38.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:37.705+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:38.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:37.706+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:39.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:38.815+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:39.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:38.815+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:39.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:38.815+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:39.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:38.816+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:39.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:38.660+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:39.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:38.660+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:39.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:38.660+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:39.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:38.661+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:39.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:38.737+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:39.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:38.737+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:39.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:38.737+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:39.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:38.738+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:40.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:39.817+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:40.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:39.817+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:40.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:39.817+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:40.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:39.817+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:40.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:39.779+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:40.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:39.779+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:40.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:39.779+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:40.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:39.780+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:40.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:39.677+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:40.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:39.677+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:40.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:39.677+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:40.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:39.678+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:41.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:40.768+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:41.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:40.768+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:41.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:40.768+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:41.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:40.769+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:41.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:40.695+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:41.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:40.695+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:41.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:40.695+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:41.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:40.696+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:41.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:40.774+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:41.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:40.774+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:41.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:40.774+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:41.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:40.775+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:41.745 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:10:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:10:41.489+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 8808 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:10:41.745 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:41.729+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:41.746 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:41.729+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:41.746 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:41.729+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:41.746 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:41.730+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:41.746 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:41.735+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:41.746 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:41.735+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:41.746 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:41.735+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:41.746 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:41.736+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:42.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:41.741+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:42.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:41.741+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:42.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:41.741+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:42.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:41.742+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:43.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:42.740+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:43.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:42.740+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:43.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:42.740+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:43.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:42.740+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:43.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:42.697+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:43.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:42.697+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:43.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:42.697+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:43.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:42.699+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:43.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:42.757+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:43.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:42.757+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:43.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:42.757+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:43.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:42.757+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:44.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:43.673+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:44.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:43.673+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:44.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:43.673+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:44.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:43.674+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:44.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:43.741+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:44.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:43.741+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:44.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:43.741+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:44.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:43.741+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:44.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:43.709+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:44.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:43.709+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:44.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:43.709+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:44.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:43.710+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:45.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:44.787+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:45.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:44.787+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:45.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:44.787+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:45.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:44.788+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:45.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:44.741+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:45.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:44.741+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:45.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:44.741+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:45.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:44.741+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:45.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:44.653+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:45.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:44.653+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:45.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:44.653+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:45.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:44.654+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:46.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:45.672+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:46.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:45.672+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:46.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:45.672+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:46.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:45.672+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:46.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:45.819+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:46.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:45.819+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:46.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:45.819+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:46.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:45.820+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:46.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:45.758+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:46.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:45.758+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:46.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:45.758+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:46.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:45.758+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:46.805 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:10:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:10:46.491+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 8901 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:10:46.806 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:46.661+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:46.806 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:46.661+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:46.806 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:46.661+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:46.806 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:46.661+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:47.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:46.863+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:47.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:46.863+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:47.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:46.863+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:47.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:46.863+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:47.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:46.802+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:47.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:46.802+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:47.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:46.802+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:47.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:46.802+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:48.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:47.842+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:48.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:47.842+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:48.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:47.843+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:48.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:47.843+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:48.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:47.669+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:48.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:47.669+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:48.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:47.669+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:48.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:47.669+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:48.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:47.872+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:48.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:47.872+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:48.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:47.872+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:48.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:47.874+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:48.896 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:48.624+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:48.896 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:48.624+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:48.896 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:48.624+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:48.896 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:48.625+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:49.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:48.904+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:49.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:48.904+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:49.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:48.904+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:49.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:48.904+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:49.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:48.892+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:49.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:48.892+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:49.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:48.892+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:49.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:48.893+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:49.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:49.622+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:49.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:49.622+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:49.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:49.622+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:49.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:49.623+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:50.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:49.933+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:50.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:49.933+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:50.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:49.933+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:50.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:49.934+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:50.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:49.919+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:50.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:49.919+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:50.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:49.919+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:50.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:49.919+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:50.900 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:50.574+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:50.900 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:50.574+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:50.900 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:50.574+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:50.900 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:50.574+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:51.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:50.931+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:51.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:50.931+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:51.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:50.931+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:51.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:50.932+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:51.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:50.896+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:51.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:50.896+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:51.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:50.896+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:51.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:50.897+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:51.866 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:10:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:10:51.492+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 8991 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:10:51.866 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:51.611+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:51.867 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:51.611+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:51.867 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:51.611+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:51.867 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:51.611+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:52.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:51.956+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:52.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:51.956+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:52.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:51.956+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:52.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:51.957+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:52.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:51.862+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:52.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:51.862+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:52.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:51.862+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:52.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:51.863+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:52.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:52.571+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:52.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:52.571+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:52.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:52.571+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:52.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:52.572+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:52.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:52.815+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:52.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:52.815+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:52.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:52.815+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:52.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:52.816+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:53.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:52.911+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:53.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:52.911+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:53.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:52.911+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:53.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:52.911+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:53.779 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:53.559+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:53.780 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:53.559+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:53.780 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:53.559+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:53.780 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:53.560+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:54.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:53.775+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:54.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:53.775+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:54.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:53.775+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:54.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:53.776+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:54.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:53.956+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:54.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:53.956+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:54.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:53.956+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:54.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:53.957+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:54.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:54.738+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:54.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:54.738+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:54.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:54.738+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:54.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:54.738+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:54.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:54.524+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:54.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:54.524+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:54.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:54.524+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:54.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:54.526+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:55.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:54.995+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:55.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:54.995+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:55.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:54.995+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:55.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:54.996+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:55.780 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:55.513+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:55.780 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:55.513+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:55.780 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:55.514+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:55.780 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:55.514+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:56.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:55.775+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:56.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:55.776+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:56.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:55.776+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:56.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:55.776+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:56.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:55.990+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:56.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:55.990+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:56.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:55.990+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:56.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:55.991+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:56.825 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:56.499+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:56.826 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:56.499+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:56.826 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:56.499+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:56.826 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:56.500+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:56.826 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:10:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:10:56.493+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 9077 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:10:57.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:56.821+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:57.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:56.821+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:57.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:56.821+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:57.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:56.822+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:57.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:57.020+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:57.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:57.020+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:57.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:57.020+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:57.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:57.021+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:57.861 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:57.540+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:57.862 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:57.540+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:57.862 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:57.540+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:57.862 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:57.541+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:58.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:57.857+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:58.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:57.857+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:58.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:57.857+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:58.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:57.858+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:58.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:58.003+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:58.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:58.003+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:58.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:58.003+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:58.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:58.004+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:58.878 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:58.584+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:58.878 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:58.584+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:58.878 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:58.584+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:58.878 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:58.584+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:59.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:58.873+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:59.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:58.873+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:59.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:58.873+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:10:59.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:58.874+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:10:59.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:59.033+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:59.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:59.033+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:59.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:59.033+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:10:59.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:10:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:10:59.034+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:10:59.830 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:59.545+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:59.830 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:59.545+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:59.831 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:59.545+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:10:59.831 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:10:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:10:59.546+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:00.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:59.826+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:00.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:59.826+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:00.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:59.826+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:00.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:10:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:10:59.827+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:00.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:00.064+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:00.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:00.064+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:00.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:00.064+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:00.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:00.064+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:00.839 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:00.538+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:00.839 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:00.538+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:00.839 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:00.538+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:00.839 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:00.539+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:01.098 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:00.835+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:01.099 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:00.835+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:01.099 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:00.835+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:01.099 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:00.836+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:01.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:01.094+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:01.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:01.094+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:01.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:01.094+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:01.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:01.095+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:01.798 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:11:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:11:01.493+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 9166 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:11:01.798 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:01.508+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:01.798 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:01.508+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:01.798 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:01.508+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:01.799 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:01.509+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:02.114 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:01.793+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:02.114 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:01.793+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:02.114 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:01.793+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:02.114 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:01.795+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:02.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:02.109+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:02.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:02.109+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:02.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:02.109+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:02.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:02.110+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:02.833 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:02.526+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:02.833 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:02.526+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:02.833 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:02.526+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:02.833 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:02.527+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:03.091 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:02.828+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:03.091 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:02.828+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:03.092 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:02.828+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:03.092 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:02.830+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:03.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:03.086+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:03.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:03.086+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:03.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:03.086+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:03.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:03.087+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:03.800 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:03.505+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:03.800 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:03.505+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:03.800 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:03.505+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:03.800 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:03.505+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:04.062 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:03.795+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:04.063 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:03.795+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:04.063 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:03.795+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:04.063 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:03.796+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:04.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:04.057+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:04.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:04.057+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:04.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:04.057+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:04.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:04.058+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:04.813 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:04.460+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:04.813 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:04.460+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:04.813 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:04.460+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:04.813 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:04.461+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:05.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:04.809+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:05.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:04.809+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:05.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:04.809+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:05.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:04.809+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:05.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:05.022+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:05.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:05.022+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:05.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:05.022+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:05.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:05.023+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:05.773 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:05.418+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:05.773 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:05.418+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:05.773 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:05.418+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:05.773 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:05.418+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:06.075 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:05.768+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:06.076 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:05.768+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:06.076 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:05.768+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:06.076 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:05.768+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:06.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:06.070+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:06.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:06.070+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:06.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:06.070+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:06.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:06.071+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:06.747 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:11:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:11:06.494+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 9249 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:11:06.748 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:06.461+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:06.748 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:06.461+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:06.748 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:06.461+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:06.748 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:06.461+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:07.096 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:06.743+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:07.096 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:06.743+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:07.096 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:06.743+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:07.096 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:06.744+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:07.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:07.090+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:07.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:07.090+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:07.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:07.090+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:07.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:07.091+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:07.749 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:07.475+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:07.749 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:07.475+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:07.749 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:07.475+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:07.749 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:07.476+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:08.074 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:07.744+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:08.074 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:07.744+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:08.074 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:07.744+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:08.074 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:07.745+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:08.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:08.070+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:08.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:08.070+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:08.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:08.070+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:08.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:08.070+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:08.709 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:08.428+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:08.709 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:08.428+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:08.709 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:08.428+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:08.710 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:08.430+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:08.710 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:08.704+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:08.710 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:08.704+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:08.710 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:08.704+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:09.099 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:08.706+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:09.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:09.095+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:09.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:09.095+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:09.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:09.095+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:09.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:09.096+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:09.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:09.403+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:09.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:09.403+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:09.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:09.403+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:09.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:09.403+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:10.054 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:09.683+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:10.054 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:09.683+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:10.055 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:09.683+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:10.055 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:09.684+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:10.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:10.050+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:10.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:10.050+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:10.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:10.050+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:10.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:10.051+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:10.669 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:10.414+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:10.669 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:10.414+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:10.669 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:10.414+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:10.669 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:10.415+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:11.055 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:10.665+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:11.055 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:10.665+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:11.055 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:10.665+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:11.055 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:10.665+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:11.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:11.101+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:11.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:11.101+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:11.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:11.101+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:11.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:11.101+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:11.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:11:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:11:11.496+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 9339 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:11:11.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:11.447+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:11.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:11.447+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:11.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:11.447+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:11.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:11.447+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:11.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:11.692+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:11.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:11.692+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:11.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:11.692+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:11.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:11.693+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:12.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:12.086+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:12.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:12.086+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:12.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:12.086+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:12.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:12.087+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:12.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:12.487+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:12.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:12.487+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:12.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:12.487+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:12.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:12.488+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:12.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:12.651+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:12.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:12.651+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:12.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:12.651+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:12.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:12.652+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:13.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:13.087+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:13.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:13.087+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:13.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:13.087+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:13.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:13.087+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:13.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:13.605+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:13.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:13.605+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:13.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:13.605+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:13.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:13.605+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:13.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:13.460+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:13.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:13.460+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:13.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:13.460+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:13.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:13.461+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:14.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:14.070+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:14.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:14.070+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:14.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:14.070+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:14.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:14.071+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:14.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:14.439+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:14.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:14.439+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:14.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:14.439+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:14.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:14.440+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:14.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:14.652+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:14.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:14.652+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:14.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:14.652+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:14.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:14.653+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:15.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:15.086+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:15.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:15.086+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:15.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:15.086+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:15.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:15.087+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:15.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:15.447+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:15.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:15.447+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:15.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:15.447+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:15.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:15.448+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:15.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:15.634+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:15.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:15.634+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:15.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:15.634+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:15.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:15.635+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:16.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:16.048+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:16.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:16.048+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:16.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:16.048+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:16.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:16.049+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:16.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:11:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:11:16.498+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 9429 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:11:16.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:16.485+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:16.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:16.485+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:16.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:16.485+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:16.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:16.486+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:16.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:16.657+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:16.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:16.657+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:16.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:16.657+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:16.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:16.657+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:17.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:17.079+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:17.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:17.079+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:17.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:17.079+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:17.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:17.080+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:17.678 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:17.469+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:17.678 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:17.469+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:17.678 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:17.469+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:17.678 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:17.469+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:18.056 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:17.674+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:18.056 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:17.674+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:18.056 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:17.674+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:18.056 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:17.675+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:18.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:18.051+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:18.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:18.051+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:18.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:18.051+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:18.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:18.053+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:18.691 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:18.436+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:18.691 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:18.436+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:18.691 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:18.436+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:18.691 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:18.437+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:19.008 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:18.687+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:19.008 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:18.687+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:19.008 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:18.687+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:19.009 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:18.687+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:19.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:19.004+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:19.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:19.004+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:19.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:19.004+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:19.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:19.005+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:19.722 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:19.463+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:19.722 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:19.463+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:19.722 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:19.463+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:19.722 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:19.464+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:20.014 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:19.717+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:20.014 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:19.717+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:20.014 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:19.717+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:20.014 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:19.719+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:20.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:20.010+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:20.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:20.010+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:20.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:20.010+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:20.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:20.011+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:20.682 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:20.423+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:20.682 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:20.423+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:20.682 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:20.423+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:20.682 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:20.424+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:20.682 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:20.678+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:20.682 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:20.678+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:20.682 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:20.678+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:21.027 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:20.679+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:21.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:21.023+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:21.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:21.023+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:21.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:21.023+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:21.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:21.024+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:21.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:11:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:11:21.499+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 9521 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:11:21.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:21.472+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:21.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:21.472+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:21.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:21.472+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:21.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:21.473+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:21.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:21.700+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:21.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:21.700+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:21.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:21.700+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:21.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:21.701+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:22.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:22.026+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:22.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:22.026+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:22.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:22.026+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:22.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:22.027+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:22.743 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:22.432+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:22.743 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:22.432+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:22.743 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:22.432+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:22.743 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:22.433+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:23.020 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:22.739+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:23.020 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:22.739+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:23.020 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:22.739+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:23.020 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:22.739+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:23.343 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:23.016+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:23.343 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:23.016+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:23.343 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:23.016+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:23.344 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:23.017+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:23.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:23.389+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:23.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:23.389+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:23.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:23.389+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:23.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:23.390+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:24.017 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:23.757+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:24.017 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:23.757+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:24.018 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:23.757+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:24.018 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:23.758+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:24.347 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:24.013+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:24.347 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:24.013+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:24.347 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:24.013+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:24.347 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:24.014+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:24.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:24.343+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:24.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:24.343+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:24.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:24.343+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:24.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:24.344+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:25.022 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:24.713+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:25.022 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:24.713+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:25.022 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:24.713+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:25.022 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:24.713+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:25.306 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:25.017+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:25.306 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:25.017+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:25.306 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:25.017+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:25.306 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:25.019+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:25.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:25.302+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:25.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:25.302+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:25.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:25.302+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:25.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:25.303+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:25.976 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:25.720+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:25.976 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:25.720+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:25.976 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:25.720+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:25.977 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:25.721+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:26.339 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:25.972+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:26.339 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:25.972+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:26.339 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:25.972+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:26.339 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:25.973+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:26.630 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:11:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:11:26.500+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 9613 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:11:26.631 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:26.334+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:26.631 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:26.334+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:26.631 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:26.334+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:26.631 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:26.335+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:26.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:26.692+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:26.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:26.692+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:26.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:26.692+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:26.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:26.693+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:27.354 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:26.952+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:27.354 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:26.952+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:27.354 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:26.952+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:27.354 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:26.953+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:27.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:27.350+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:27.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:27.350+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:27.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:27.350+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:27.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:27.351+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:28.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:27.921+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:28.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:27.921+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:28.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:27.921+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:28.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:27.922+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:28.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:27.679+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:28.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:27.679+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:28.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:27.679+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:28.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:27.679+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:28.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:28.374+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:28.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:28.374+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:28.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:28.374+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:28.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:28.375+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:28.972 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:28.685+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:28.972 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:28.685+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:28.972 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:28.685+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:28.972 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:28.685+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:29.380 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:28.968+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:29.380 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:28.968+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:29.380 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:28.968+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:29.380 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:28.969+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:29.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:29.376+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:29.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:29.376+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:29.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:29.376+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:29.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:29.377+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:30.012 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:29.678+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:30.012 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:29.678+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:30.012 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:29.678+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:30.013 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:29.679+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:30.355 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:30.008+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:30.355 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:30.008+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:30.355 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:30.008+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:30.355 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:30.009+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:30.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:30.351+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:30.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:30.351+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:30.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:30.351+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:30.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:30.352+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:31.063 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:30.728+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:31.063 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:30.728+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:31.063 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:30.728+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:31.063 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:30.729+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:31.379 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:31.059+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:31.379 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:31.059+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:31.379 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:31.059+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:31.379 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:31.059+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:31.631 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:11:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:11:31.501+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 9710 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:11:31.631 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:31.375+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:31.631 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:31.375+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:31.631 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:31.375+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:31.631 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:31.375+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:31.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:31.729+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:31.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:31.729+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:31.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:31.729+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:31.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:31.730+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:32.362 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:32.037+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:32.362 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:32.037+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:32.362 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:32.037+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:32.363 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:32.038+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:32.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:32.358+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:32.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:32.358+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:32.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:32.358+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:32.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:32.359+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:33.077 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:32.774+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:33.077 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:32.774+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:33.077 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:32.774+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:33.077 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:32.774+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:33.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:33.073+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:33.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:33.073+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:33.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:33.073+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:33.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:33.074+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:33.765 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:33.408+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:33.765 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:33.408+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:33.765 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:33.408+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:33.765 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:33.409+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:34.032 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:33.761+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:34.032 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:33.761+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:34.032 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:33.761+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:34.032 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:33.762+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:34.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:34.028+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:34.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:34.028+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:34.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:34.028+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:34.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:34.029+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:34.748 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:34.437+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:34.748 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:34.437+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:34.748 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:34.437+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:34.748 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:34.438+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:35.081 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:34.744+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:35.081 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:34.744+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:35.081 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:34.744+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:35.081 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:34.744+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:35.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:35.077+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:35.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:35.077+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:35.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:35.077+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:35.078+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:35.770 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:35.437+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:35.771 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:35.437+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:35.771 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:35.437+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:35.771 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:35.438+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:36.063 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:35.766+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:36.063 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:35.766+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:36.063 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:35.766+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:36.063 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:35.768+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:36.404 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:36.059+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:36.404 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:36.059+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:36.404 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:36.059+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:36.404 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:36.060+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:36.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:11:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:11:36.503+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 9804 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:11:36.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:36.400+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:36.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:36.400+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:36.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:36.400+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:36.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:36.401+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:37.076 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:36.765+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:37.076 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:36.765+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:37.076 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:36.765+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:37.076 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:36.766+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:37.390 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:37.072+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:37.390 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:37.072+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:37.390 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:37.072+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:37.390 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:37.072+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:37.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:37.386+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:37.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:37.386+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:37.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:37.386+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:37.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:37.387+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:38.112 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:37.811+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:38.112 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:37.811+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:38.112 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:37.811+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:38.112 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:37.811+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:38.404 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:38.108+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:38.404 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:38.108+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:38.404 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:38.108+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:38.404 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:38.109+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:38.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:38.399+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:38.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:38.400+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:38.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:38.400+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:38.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:38.400+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:39.121 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:38.839+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:39.122 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:38.839+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:39.122 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:38.839+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:39.122 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:38.839+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:39.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:39.117+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:39.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:39.117+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:39.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:39.117+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:39.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:39.118+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:39.802 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:39.416+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:39.802 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:39.416+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:39.802 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:39.416+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:39.802 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:39.417+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:40.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:39.798+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:40.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:39.798+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:40.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:39.798+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:40.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:39.799+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:40.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:40.167+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:40.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:40.167+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:40.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:40.167+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:40.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:40.168+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:40.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:40.382+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:40.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:40.382+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:40.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:40.382+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:40.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:40.383+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:41.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:40.793+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:41.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:40.793+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:41.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:40.793+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:41.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:40.794+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:41.508 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:41.381+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:41.508 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:41.381+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:41.508 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:41.381+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:41.511 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:41.381+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:41.511 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:41.216+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:41.511 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:41.216+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:41.511 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:41.216+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:41.511 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:41.216+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:41.764 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:11:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:11:41.504+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 9900 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:11:42.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:41.760+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:42.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:41.760+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:42.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:41.760+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:42.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:41.761+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:42.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:42.402+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:42.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:42.402+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:42.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:42.402+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:42.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:42.403+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:42.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:42.172+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:42.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:42.172+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:42.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:42.172+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:42.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:42.172+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:43.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:42.740+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:43.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:42.740+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:43.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:42.740+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:43.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:42.740+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:43.452 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:43.186+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:43.452 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:43.186+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:43.452 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:43.186+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:43.453 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:43.187+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:43.761 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:43.448+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:43.761 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:43.448+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:43.761 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:43.448+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:43.761 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:43.449+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:44.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:43.757+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:44.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:43.757+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:44.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:43.757+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:44.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:43.758+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:44.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:44.447+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:44.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:44.447+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:44.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:44.447+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:44.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:44.448+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:44.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:44.227+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:44.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:44.227+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:44.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:44.227+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:44.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:44.228+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:45.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:44.719+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:45.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:44.719+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:45.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:44.719+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:45.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:44.720+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:45.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:45.185+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:45.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:45.185+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:45.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:45.185+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:45.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:45.186+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:45.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:45.421+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:45.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:45.421+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:45.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:45.421+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:45.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:45.422+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:46.142 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:45.754+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:46.142 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:45.754+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:46.142 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:45.754+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:46.142 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:45.755+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:46.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:46.138+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:46.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:46.138+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:46.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:46.138+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:46.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:46.139+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:46.729 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:11:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:11:46.505+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 9994 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:11:46.729 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:46.455+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:46.729 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:46.455+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:46.729 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:46.455+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:46.729 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:46.456+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:47.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:46.725+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:47.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:46.725+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:47.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:46.725+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:47.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:46.726+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:47.443 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:47.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:47.443 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:47.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:47.444 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:47.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:47.444 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:47.171+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:47.706 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:47.439+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:47.706 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:47.439+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:47.706 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:47.439+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:47.707 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:47.440+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:48.142 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:47.702+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:48.142 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:47.702+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:48.142 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:47.702+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:48.142 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:47.703+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:48.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:48.138+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:48.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:48.138+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:48.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:48.138+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:48.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:48.139+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:48.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:48.485+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:48.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:48.485+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:48.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:48.485+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:48.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:48.486+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:48.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:48.670+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:48.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:48.670+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:48.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:48.670+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:48.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:48.671+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:49.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:49.151+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:49.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:49.151+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:49.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:49.151+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:49.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:49.152+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:49.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:49.661+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:49.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:49.661+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:49.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:49.661+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:49.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:49.661+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:49.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:49.465+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:49.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:49.465+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:49.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:49.465+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:49.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:49.465+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:50.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:50.144+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:50.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:50.144+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:50.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:50.144+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:50.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:50.145+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:50.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:50.504+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:50.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:50.504+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:50.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:50.504+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:50.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:50.505+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:50.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:50.618+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:50.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:50.618+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:50.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:50.618+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:50.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:50.619+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:51.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:51.140+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:51.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:51.140+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:51.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:51.140+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:51.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:51.141+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:51.679 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:51.468+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:51.679 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:51.468+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:51.679 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:51.468+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:51.679 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:51.468+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:51.680 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:11:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:11:51.506+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 10091 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:11:51.680 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:51.622+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:51.680 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:51.622+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:51.680 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:51.622+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:51.680 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:51.623+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:52.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:52.124+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:52.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:52.124+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:52.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:52.124+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:52.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:52.125+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:52.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:52.477+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:52.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:52.477+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:52.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:52.477+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:52.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:52.478+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:52.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:52.591+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:52.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:52.591+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:52.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:52.591+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:52.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:52.592+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:53.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:53.116+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:53.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:53.116+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:53.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:53.116+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:53.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:53.117+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:53.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:53.521+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:53.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:53.521+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:53.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:53.521+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:53.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:53.522+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:53.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:53.603+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:53.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:53.603+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:53.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:53.603+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:53.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:53.604+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:54.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:54.141+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:54.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:54.141+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:54.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:54.141+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:54.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:54.142+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:54.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:54.650+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:54.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:54.650+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:54.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:54.650+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:54.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:54.651+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:54.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:54.521+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:54.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:54.521+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:54.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:54.521+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:54.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:54.522+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:55.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:55.117+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:55.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:55.117+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:55.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:55.117+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:55.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:55.118+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:55.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:55.516+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:55.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:55.516+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:55.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:55.516+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:55.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:55.517+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:55.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:55.622+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:55.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:55.622+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:55.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:55.622+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:55.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:55.622+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:56.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:56.126+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:56.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:56.126+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:56.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:56.126+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:56.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:56.127+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:56.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:11:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:11:56.507+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 10186 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:11:56.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:56.612+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:56.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:56.612+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:56.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:56.612+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:56.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:56.613+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:56.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:56.475+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:56.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:56.475+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:56.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:56.475+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:56.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:56.476+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:57.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:57.113+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:57.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:57.113+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:57.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:57.113+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:57.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:57.113+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:57.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:57.438+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:57.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:57.438+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:57.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:57.438+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:57.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:57.438+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:57.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:57.572+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:57.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:57.572+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:57.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:57.572+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:57.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:57.573+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:58.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:58.145+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:58.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:58.145+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:58.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:58.145+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:58.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:58.146+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:58.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:58.466+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:58.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:58.466+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:58.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:58.466+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:58.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:58.466+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:58.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:58.531+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:58.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:58.531+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:58.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:58.531+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:58.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:58.532+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:11:59.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:59.123+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:59.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:59.123+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:59.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:59.123+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:11:59.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:11:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:11:59.124+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:59.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:59.498+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:59.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:59.498+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:59.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:59.498+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:11:59.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:11:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:11:59.499+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:11:59.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:59.492+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:59.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:59.492+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:59.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:59.492+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:11:59.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:11:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:11:59.493+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:00.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:00.078+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:00.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:00.078+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:00.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:00.078+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:00.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:00.079+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:00.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:00.478+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:00.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:00.478+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:00.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:00.478+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:00.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:00.478+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:00.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:00.461+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:00.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:00.461+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:00.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:00.461+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:00.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:00.462+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:01.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:01.060+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:01.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:01.060+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:01.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:01.060+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:01.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:01.061+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:01.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:12:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:12:01.509+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 10283 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:12:01.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:01.499+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:01.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:01.499+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:01.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:01.499+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:01.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:01.500+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:01.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:01.455+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:01.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:01.455+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:01.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:01.455+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:01.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:01.456+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:02.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:02.036+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:02.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:02.036+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:02.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:02.036+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:02.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:02.037+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:02.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:02.468+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:02.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:02.468+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:02.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:02.468+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:02.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:02.468+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:02.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:02.485+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:02.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:02.485+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:02.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:02.485+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:02.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:02.486+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:03.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:03.041+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:03.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:03.041+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:03.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:03.041+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:03.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:03.043+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:03.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:03.513+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:03.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:03.513+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:03.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:03.514+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:03.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:03.515+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:03.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:03.518+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:03.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:03.518+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:03.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:03.518+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:03.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:03.519+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:04.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:04.079+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:04.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:04.079+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:04.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:04.079+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:04.080+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:04.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:04.560+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:04.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:04.560+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:04.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:04.560+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:04.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:04.561+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:04.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:04.540+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:04.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:04.540+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:04.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:04.540+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:04.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:04.541+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:05.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:05.047+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:05.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:05.047+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:05.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:05.047+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:05.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:05.048+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:05.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:05.523+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:05.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:05.523+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:05.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:05.523+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:05.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:05.524+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:05.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:05.531+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:05.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:05.531+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:05.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:05.531+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:05.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:05.532+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:06.350 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:06.059+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:06.350 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:06.059+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:06.350 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:06.059+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:06.350 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:06.060+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:06.639 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:12:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:12:06.510+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 10376 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:12:06.640 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:06.537+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:06.640 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:06.537+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:06.640 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:06.537+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:06.640 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:06.538+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:06.640 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:06.489+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:06.640 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:06.489+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:06.641 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:06.489+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:06.641 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:06.490+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:07.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:07.019+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:07.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:07.019+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:07.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:07.019+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:07.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:07.020+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:07.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:07.491+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:07.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:07.491+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:07.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:07.491+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:07.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:07.492+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:07.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:07.457+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:07.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:07.457+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:07.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:07.457+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:07.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:07.458+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:08.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:08.043+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:08.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:08.043+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:08.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:08.043+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:08.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:08.044+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:08.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:08.453+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:08.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:08.453+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:08.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:08.453+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:08.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:08.454+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:08.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:08.411+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:08.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:08.411+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:08.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:08.411+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:08.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:08.412+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:09.384 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:09.001+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:09.384 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:09.001+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:09.385 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:09.001+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:09.385 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:09.002+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:09.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:09.470+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:09.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:09.470+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:09.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:09.470+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:09.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:09.470+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:09.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:09.380+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:09.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:09.380+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:09.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:09.380+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:09.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:09.381+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:10.402 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:09.956+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:10.402 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:09.956+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:10.402 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:09.956+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:10.403 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:09.957+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:10.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:10.398+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:10.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:10.398+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:10.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:10.398+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:10.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:10.399+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:10.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:10.493+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:10.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:10.493+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:10.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:10.493+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:10.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:10.494+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:11.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:10.975+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:11.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:10.975+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:11.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:10.975+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:11.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:10.976+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:11.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:12:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:12:11.511+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 10472 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:12:11.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:11.458+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:11.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:11.458+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:11.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:11.458+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:11.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:11.459+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:11.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:11.448+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:11.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:11.448+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:11.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:11.448+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:11.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:11.448+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:12.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:12.003+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:12.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:12.003+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:12.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:12.003+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:12.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:12.004+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:12.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:12.491+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:12.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:12.491+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:12.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:12.491+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:12.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:12.492+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:12.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:12.425+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:12.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:12.425+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:12.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:12.425+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:12.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:12.426+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:13.383 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:12.989+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:13.383 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:12.989+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:13.383 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:12.989+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:13.383 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:12.990+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:13.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:13.453+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:13.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:13.453+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:13.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:13.453+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:13.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:13.454+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:13.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:13.379+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:13.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:13.379+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:13.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:13.379+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:13.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:13.380+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:14.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:13.957+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:14.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:13.957+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:14.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:13.957+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:14.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:13.958+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:14.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:14.423+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:14.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:14.423+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:14.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:14.423+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:14.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:14.424+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:14.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:14.427+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:14.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:14.427+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:14.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:14.427+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:14.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:14.428+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:15.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:14.941+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:15.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:14.941+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:15.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:14.941+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:15.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:14.942+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:15.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:15.438+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:15.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:15.438+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:15.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:15.438+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:15.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:15.439+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:15.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:15.455+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:15.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:15.455+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:15.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:15.455+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:15.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:15.456+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:16.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:15.945+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:16.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:15.945+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:16.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:15.945+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:16.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:15.946+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:16.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:12:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:12:16.512+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 10565 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:12:16.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:16.470+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:16.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:16.470+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:16.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:16.470+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:16.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:16.470+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:16.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:16.417+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:16.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:16.417+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:16.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:16.417+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:16.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:16.417+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:17.402 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:16.907+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:17.403 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:16.907+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:17.403 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:16.907+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:17.403 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:16.908+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:17.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:17.460+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:17.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:17.460+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:17.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:17.460+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:17.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:17.461+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:17.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:17.398+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:17.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:17.398+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:17.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:17.398+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:17.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:17.400+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:18.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:17.883+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:18.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:17.883+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:18.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:17.883+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:18.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:17.884+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:18.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:18.437+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:18.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:18.437+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:18.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:18.437+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:18.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:18.438+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:18.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:18.369+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:18.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:18.369+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:18.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:18.369+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:18.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:18.370+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:19.384 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:18.911+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:19.384 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:18.911+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:19.384 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:18.911+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:19.384 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:18.911+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:19.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:19.460+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:19.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:19.460+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:19.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:19.460+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:19.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:19.461+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:19.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:19.380+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:19.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:19.380+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:19.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:19.380+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:19.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:19.381+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:20.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:19.944+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:20.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:19.944+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:20.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:19.944+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:20.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:19.945+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:20.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:20.461+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:20.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:20.461+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:20.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:20.461+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:20.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:20.461+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:20.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:20.429+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:20.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:20.429+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:20.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:20.429+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:20.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:20.430+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:21.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:20.971+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:21.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:20.971+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:21.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:20.971+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:21.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:20.972+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:21.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:12:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:12:21.513+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 10660 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:12:21.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:21.502+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:21.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:21.502+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:21.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:21.502+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:21.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:21.502+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:21.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:21.463+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:21.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:21.463+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:21.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:21.463+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:21.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:21.464+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:22.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:21.963+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:22.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:21.963+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:22.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:21.963+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:22.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:21.964+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:22.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:22.518+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:22.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:22.518+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:22.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:22.518+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:22.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:22.519+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:22.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:22.475+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:22.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:22.475+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:22.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:22.475+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:22.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:22.476+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:23.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:22.921+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:23.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:22.921+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:23.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:22.921+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:23.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:22.922+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:23.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:23.504+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:23.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:23.504+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:23.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:23.504+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:23.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:23.505+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:23.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:23.435+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:23.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:23.435+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:23.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:23.435+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:23.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:23.436+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:24.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:23.964+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:24.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:23.964+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:24.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:23.964+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:24.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:23.965+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:24.782 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:24.503+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:24.782 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:24.503+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:24.782 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:24.503+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:24.782 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:24.504+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:24.782 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:24.481+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:24.782 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:24.481+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:24.783 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:24.481+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:24.783 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:24.482+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:25.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:24.970+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:25.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:24.970+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:25.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:24.970+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:25.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:24.971+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:25.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:25.466+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:25.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:25.466+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:25.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:25.466+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:25.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:25.467+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:25.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:25.443+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:25.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:25.443+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:25.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:25.443+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:25.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:25.444+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:26.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:25.968+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:26.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:25.968+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:26.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:25.968+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:25.968+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:26.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:12:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:12:26.515+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 10751 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:12:26.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:26.413+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:26.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:26.413+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:26.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:26.413+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:26.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:26.414+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:26.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:26.429+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:26.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:26.429+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:26.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:26.429+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:26.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:26.430+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:27.378 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:26.966+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:27.378 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:26.966+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:27.379 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:26.966+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:27.379 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:26.967+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:27.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:27.440+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:27.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:27.440+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:27.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:27.440+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:27.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:27.441+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:27.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:27.374+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:27.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:27.374+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:27.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:27.374+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:27.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:27.375+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:28.353 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:28.004+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:28.354 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:28.004+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:28.354 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:28.004+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:28.354 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:28.005+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:28.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:28.349+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:28.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:28.349+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:28.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:28.349+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:28.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:28.350+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:28.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:28.472+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:28.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:28.472+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:28.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:28.472+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:28.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:28.473+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:29.351 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:29.020+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:29.351 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:29.020+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:29.351 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:29.020+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:29.351 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:29.022+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:29.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:29.505+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:29.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:29.505+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:29.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:29.505+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:29.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:29.506+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:29.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:29.347+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:29.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:29.347+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:29.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:29.347+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:29.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:29.348+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:30.369 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:29.988+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:30.369 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:29.988+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:30.369 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:29.988+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:30.369 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:29.989+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:30.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:30.365+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:30.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:30.365+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:30.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:30.365+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:30.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:30.366+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:30.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:30.542+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:30.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:30.542+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:30.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:30.542+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:30.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:30.543+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:31.335 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:30.968+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:31.336 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:30.968+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:31.336 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:30.968+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:31.336 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:30.968+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:31.646 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:12:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:12:31.516+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 10847 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:12:31.646 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:31.529+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:31.646 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:31.529+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:31.646 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:31.529+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:31.646 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:31.530+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:31.646 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:31.331+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:31.646 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:31.331+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:31.647 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:31.331+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:31.647 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:31.332+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:32.299 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:31.977+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:32.299 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:31.977+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:32.299 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:31.977+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:32.299 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:31.978+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:32.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:32.527+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:32.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:32.527+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:32.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:32.527+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:32.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:32.528+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:32.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:32.294+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:32.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:32.294+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:32.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:32.294+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:32.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:32.295+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:33.330 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:33.024+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:33.330 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:33.024+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:33.331 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:33.024+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:33.331 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:33.025+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:33.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:33.509+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:33.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:33.509+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:33.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:33.509+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:33.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:33.510+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:33.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:33.326+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:33.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:33.326+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:33.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:33.326+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:33.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:33.327+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:34.285 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:34.019+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:34.285 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:34.019+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:34.285 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:34.019+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:34.285 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:34.019+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:34.548 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:34.281+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:34.548 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:34.281+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:34.548 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:34.281+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:34.548 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:34.282+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:34.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:34.543+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:34.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:34.543+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:34.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:34.543+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:34.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:34.544+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:35.314 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:35.031+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:35.314 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:35.031+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:35.314 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:35.031+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:35.315 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:35.031+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:35.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:35.507+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:35.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:35.507+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:35.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:35.507+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:35.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:35.507+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:35.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:35.310+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:35.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:35.310+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:35.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:35.310+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:35.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:35.311+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:36.364 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:36.061+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:36.364 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:36.061+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:36.364 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:36.061+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:36.364 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:36.061+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:36.647 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:12:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:12:36.517+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 10941 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:12:36.648 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:36.461+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:36.648 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:36.461+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:36.648 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:36.461+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:36.648 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:36.462+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:36.648 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:36.360+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:36.648 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:36.360+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:36.649 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:36.360+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:36.649 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:36.361+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:37.344 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:37.035+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:37.345 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:37.035+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:37.345 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:37.035+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:37.345 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:37.036+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:37.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:37.455+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:37.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:37.455+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:37.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:37.455+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:37.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:37.456+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:37.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:37.340+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:37.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:37.340+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:37.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:37.340+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:37.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:37.341+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:38.348 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:38.060+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:38.348 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:38.060+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:38.348 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:38.060+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:38.348 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:38.060+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:38.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:38.469+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:38.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:38.469+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:38.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:38.469+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:38.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:38.470+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:38.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:38.343+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:38.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:38.343+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:38.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:38.343+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:38.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:38.344+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:39.341 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:39.025+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:39.341 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:39.025+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:39.342 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:39.025+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:39.342 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:39.026+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:39.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:39.497+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:39.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:39.497+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:39.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:39.497+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:39.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:39.498+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:39.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:39.336+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:39.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:39.336+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:39.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:39.336+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:39.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:39.337+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:40.312 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:40.036+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:40.312 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:40.036+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:40.312 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:40.036+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:40.312 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:40.036+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:40.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:40.545+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:40.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:40.545+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:40.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:40.545+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:40.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:40.546+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:40.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:40.307+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:40.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:40.307+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:40.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:40.307+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:40.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:40.308+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:41.320 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:41.005+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:41.320 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:41.005+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:41.320 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:41.005+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:41.320 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:41.006+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:41.576 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:12:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:12:41.517+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 11037 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:12:41.576 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:41.315+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:41.577 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:41.315+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:41.577 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:41.315+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:41.577 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:41.316+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:41.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:41.571+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:41.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:41.571+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:41.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:41.571+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:41.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:41.572+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:42.323 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:41.965+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:42.324 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:41.965+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:42.324 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:41.965+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:42.324 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:41.967+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:42.604 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:42.318+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:42.604 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:42.318+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:42.604 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:42.318+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:42.604 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:42.319+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:42.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:42.599+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:42.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:42.599+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:42.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:42.599+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:42.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:42.599+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:43.214 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:42.937+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:43.214 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:42.937+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:43.214 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:42.937+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:43.215 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:42.938+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:43.498 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:43.340+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:43.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:43.340+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:43.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:43.340+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:43.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:43.340+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:43.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:43.555+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:43.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:43.555+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:43.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:43.555+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:43.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:43.555+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:44.319 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:43.926+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:44.319 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:43.926+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:44.319 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:43.926+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:44.319 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:43.927+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:44.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:44.543+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:44.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:44.543+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:44.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:44.543+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:44.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:44.544+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:44.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:44.314+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:44.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:44.314+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:44.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:44.314+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:44.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:44.315+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:45.313 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:44.904+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:45.313 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:44.904+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:45.313 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:44.904+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:45.313 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:44.904+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:45.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:45.547+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:45.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:45.547+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:45.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:45.547+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:45.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:45.548+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:45.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:45.309+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:45.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:45.309+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:45.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:45.309+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:45.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:45.310+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:46.286 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:45.953+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:46.286 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:45.953+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:46.286 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:45.953+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:46.287 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:45.954+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:46.537 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:12:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:12:46.520+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 11130 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:12:46.538 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:46.515+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:46.538 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:46.515+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:46.538 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:46.515+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:46.538 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:46.516+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:46.538 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:46.345+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:46.538 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:46.345+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:46.538 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:46.345+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:46.538 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:46.346+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:47.329 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:46.988+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:47.329 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:46.988+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:47.329 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:46.988+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:47.329 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:46.988+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:47.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:47.324+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:47.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:47.324+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:47.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:47.324+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:47.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:47.325+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:47.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:47.528+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:47.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:47.528+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:47.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:47.528+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:47.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:47.529+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:48.325 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:47.957+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:48.326 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:47.957+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:48.326 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:47.957+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:48.326 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:47.957+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:48.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:48.561+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:48.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:48.561+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:48.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:48.561+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:48.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:48.562+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:48.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:48.321+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:48.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:48.321+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:48.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:48.321+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:48.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:48.322+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:49.355 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:48.958+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:49.355 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:48.958+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:49.355 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:48.958+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:49.355 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:48.959+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:49.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:49.350+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:49.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:49.350+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:49.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:49.350+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:49.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:49.352+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:49.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:49.589+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:49.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:49.589+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:49.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:49.589+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:49.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:49.590+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:50.367 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:49.948+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:50.367 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:49.949+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:50.367 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:49.949+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:50.367 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:49.950+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:50.623 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:50.363+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:50.623 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:50.363+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:50.624 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:50.363+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:50.624 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:50.364+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:50.904 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:50.619+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:50.904 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:50.619+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:50.905 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:50.619+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:50.905 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:50.620+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:51.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:50.900+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:51.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:50.900+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:51.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:50.900+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:51.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:50.901+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:51.646 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:12:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:12:51.521+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 11225 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:12:51.646 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:51.384+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:51.646 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:51.384+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:51.647 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:51.384+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:51.647 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:51.385+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:51.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:51.642+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:51.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:51.642+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:51.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:51.642+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:51.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:51.643+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:52.370 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:51.919+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:52.370 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:51.919+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:52.370 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:51.919+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:52.370 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:51.920+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:52.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:52.603+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:52.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:52.603+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:52.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:52.603+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:52.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:52.603+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:52.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:52.365+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:52.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:52.365+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:52.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:52.365+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:52.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:52.366+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:53.396 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:52.962+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:53.396 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:52.962+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:53.396 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:52.962+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:53.397 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:52.963+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:53.649 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:53.392+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:53.649 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:53.392+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:53.649 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:53.392+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:53.649 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:53.393+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:53.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:53.645+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:53.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:53.645+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:53.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:53.645+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:53.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:53.646+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:54.366 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:53.992+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:54.366 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:53.992+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:54.366 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:53.992+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:54.366 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:53.993+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:54.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:54.361+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:54.660 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:54.361+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:54.660 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:54.361+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:54.660 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:54.362+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:54.660 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:54.609+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:54.660 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:54.609+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:54.660 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:54.609+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:54.660 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:54.610+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:55.382 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:54.953+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:55.382 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:54.953+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:55.382 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:54.953+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:55.383 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:54.954+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:55.655 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:55.378+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:55.655 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:55.378+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:55.655 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:55.378+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:55.655 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:55.379+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:55.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:55.651+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:55.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:55.651+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:55.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:55.651+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:55.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:55.651+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:56.400 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:55.985+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:56.400 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:55.985+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:56.400 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:55.985+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:56.400 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:55.986+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:56.653 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:12:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:12:56.523+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 11320 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:12:56.654 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:56.637+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:56.654 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:56.637+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:56.654 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:56.637+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:56.654 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:56.637+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:56.655 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:56.395+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:56.655 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:56.395+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:56.655 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:56.395+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:56.655 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:56.397+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:57.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:56.994+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:57.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:56.994+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:57.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:56.994+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:57.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:56.995+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:57.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:57.428+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:57.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:57.428+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:57.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:57.428+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:57.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:57.428+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:57.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:57.594+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:57.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:57.594+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:57.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:57.594+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:57.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:57.595+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:58.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:58.027+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:58.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:58.027+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:58.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:58.027+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:58.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:58.028+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:58.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:58.594+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:58.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:58.594+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:58.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:58.594+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:58.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:58.595+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:58.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:58.424+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:58.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:58.424+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:58.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:58.424+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:58.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:58.425+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:12:59.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:59.058+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:59.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:59.058+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:59.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:59.058+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:12:59.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:12:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:12:59.059+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:59.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:59.577+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:59.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:59.577+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:59.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:59.577+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:12:59.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:12:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:12:59.578+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:12:59.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:59.406+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:59.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:59.406+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:59.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:59.406+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:12:59.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:12:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:12:59.407+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:00.404 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:00.024+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:00.405 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:00.024+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:00.405 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:00.024+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:00.405 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:00.024+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:00.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:00.595+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:00.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:00.595+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:00.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:00.595+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:00.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:00.596+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:00.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:00.400+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:00.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:00.400+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:00.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:00.400+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:00.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:00.401+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:01.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:00.993+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:01.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:00.993+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:01.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:00.993+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:01.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:00.994+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:01.685 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:13:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:13:01.524+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 11414 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:13:01.685 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:01.615+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:01.685 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:01.615+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:01.685 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:01.615+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:01.685 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:01.615+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:01.685 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:01.417+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:01.686 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:01.417+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:01.686 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:01.417+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:01.686 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:01.417+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:02.406 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:02.024+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:02.406 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:02.024+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:02.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:02.024+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:02.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:02.025+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:02.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:02.402+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:02.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:02.402+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:02.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:02.402+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:02.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:02.403+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:02.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:02.611+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:02.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:02.611+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:02.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:02.611+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:02.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:02.612+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:03.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:02.978+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:03.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:02.978+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:03.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:02.978+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:03.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:02.979+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:03.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:03.651+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:03.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:03.651+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:03.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:03.651+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:03.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:03.652+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:03.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:03.421+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:03.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:03.421+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:03.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:03.421+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:03.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:03.422+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:04.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:03.959+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:04.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:03.959+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:04.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:03.959+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:03.960+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:04.704 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:04.434+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:04.704 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:04.434+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:04.704 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:04.434+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:04.704 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:04.435+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:04.984 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:04.699+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:04.984 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:04.699+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:04.984 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:04.699+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:04.984 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:04.700+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:05.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:04.980+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:05.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:04.980+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:05.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:04.980+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:05.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:04.981+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:05.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:05.691+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:05.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:05.691+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:05.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:05.691+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:05.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:05.692+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:05.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:05.465+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:05.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:05.465+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:05.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:05.465+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:05.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:05.466+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:06.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:05.982+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:06.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:05.982+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:06.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:05.982+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:06.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:05.983+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:06.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:13:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:13:06.525+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 11508 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:13:06.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:06.679+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:06.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:06.679+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:06.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:06.679+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:06.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:06.680+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:06.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:06.498+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:06.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:06.498+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:06.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:06.498+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:06.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:06.499+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:07.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:06.998+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:07.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:06.998+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:07.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:06.998+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:07.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:06.999+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:07.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:07.642+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:07.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:07.642+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:07.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:07.642+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:07.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:07.643+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:07.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:07.542+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:07.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:07.542+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:07.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:07.542+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:07.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:07.543+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:08.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:07.956+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:08.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:07.956+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:08.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:07.956+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:08.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:07.956+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:08.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:08.685+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:08.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:08.685+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:08.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:08.685+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:08.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:08.685+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:08.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:08.534+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:08.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:08.534+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:08.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:08.534+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:08.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:08.535+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:09.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:08.979+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:09.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:08.979+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:09.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:08.979+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:09.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:08.980+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:09.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:09.716+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:09.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:09.716+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:09.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:09.716+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:09.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:09.717+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:09.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:09.584+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:09.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:09.584+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:09.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:09.584+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:09.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:09.585+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:10.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:09.993+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:10.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:09.993+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:10.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:09.993+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:10.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:09.994+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:10.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:10.694+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:10.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:10.694+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:10.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:10.694+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:10.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:10.694+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:10.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:10.626+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:10.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:10.626+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:10.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:10.626+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:10.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:10.627+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:11.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:10.988+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:11.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:10.988+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:11.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:10.988+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:11.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:10.989+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:11.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:13:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:13:11.527+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 11603 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:13:11.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:11.715+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:11.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:11.715+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:11.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:11.715+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:11.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:11.716+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:11.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:11.584+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:11.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:11.584+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:11.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:11.584+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:11.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:11.584+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:12.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:11.973+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:12.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:11.973+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:12.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:11.973+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:12.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:11.975+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:12.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:12.733+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:12.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:12.733+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:12.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:12.733+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:12.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:12.733+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:12.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:12.597+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:12.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:12.597+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:12.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:12.597+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:12.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:12.597+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:13.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:12.942+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:13.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:12.942+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:13.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:12.942+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:13.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:12.943+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:13.776 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:13.585+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:13.776 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:13.585+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:13.777 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:13.585+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:13.777 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:13.585+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:14.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:13.772+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:14.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:13.772+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:14.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:13.772+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:14.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:13.773+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:14.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:13.969+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:14.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:13.969+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:14.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:13.969+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:14.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:13.970+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:14.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:14.770+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:14.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:14.770+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:14.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:14.770+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:14.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:14.771+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:14.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:14.560+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:14.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:14.560+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:14.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:14.560+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:14.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:14.561+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:15.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:14.961+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:15.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:14.961+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:15.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:14.961+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:15.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:14.961+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:15.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:15.793+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:15.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:15.793+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:15.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:15.793+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:15.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:15.794+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:15.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:15.569+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:15.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:15.569+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:15.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:15.569+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:15.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:15.570+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:16.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:15.933+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:16.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:15.933+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:16.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:15.933+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:16.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:15.933+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:16.809 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:13:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:13:16.528+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 11697 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:13:16.810 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:16.524+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:16.810 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:16.524+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:16.810 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:16.524+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:16.810 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:16.525+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:17.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:16.805+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:17.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:16.805+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:17.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:16.805+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:17.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:16.806+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:17.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:16.941+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:17.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:16.941+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:17.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:16.941+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:17.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:16.942+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:17.792 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:17.519+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:17.792 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:17.519+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:17.792 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:17.519+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:17.792 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:17.520+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:18.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:17.788+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:18.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:17.788+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:18.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:17.788+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:18.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:17.789+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:18.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:17.988+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:18.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:17.988+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:18.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:17.988+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:18.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:17.989+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:18.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:18.561+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:18.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:18.562+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:18.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:18.562+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:18.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:18.563+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:18.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:18.774+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:18.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:18.774+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:18.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:18.774+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:18.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:18.774+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:19.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:19.010+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:19.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:19.010+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:19.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:19.010+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:19.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:19.011+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:19.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:19.799+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:19.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:19.799+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:19.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:19.799+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:19.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:19.800+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:19.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:19.587+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:19.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:19.587+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:19.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:19.587+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:19.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:19.588+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:20.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:20.027+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:20.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:20.027+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:20.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:20.027+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:20.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:20.027+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:20.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:20.827+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:20.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:20.827+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:20.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:20.827+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:20.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:20.828+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:20.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:20.627+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:20.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:20.627+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:20.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:20.627+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:20.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:20.628+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:21.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:21.067+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:21.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:21.067+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:21.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:21.067+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:21.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:21.068+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:21.845 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:13:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:13:21.529+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 11791 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:13:21.845 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:21.664+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:21.845 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:21.664+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:21.845 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:21.664+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:21.845 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:21.664+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:22.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:21.841+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:22.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:21.841+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:22.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:21.841+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:22.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:21.842+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:22.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:22.070+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:22.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:22.070+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:22.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:22.070+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:22.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:22.071+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:22.876 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:22.617+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:22.876 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:22.617+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:22.876 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:22.617+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:22.876 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:22.618+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:23.150 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:22.872+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:23.150 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:22.872+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:23.150 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:22.872+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:23.150 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:22.873+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:23.150 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:23.091+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:23.151 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:23.091+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:23.151 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:23.091+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:23.151 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:23.092+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:23.846 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:23.594+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:23.846 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:23.594+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:23.846 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:23.594+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:23.846 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:23.595+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:24.116 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:23.858+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:24.116 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:23.858+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:24.116 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:23.858+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:24.117 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:23.859+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:24.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:24.112+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:24.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:24.112+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:24.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:24.112+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:24.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:24.113+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:24.827 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:24.570+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:24.827 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:24.570+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:24.827 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:24.570+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:24.827 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:24.572+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:25.153 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:24.823+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:25.154 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:24.823+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:25.154 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:24.823+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:25.154 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:24.824+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:25.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:25.149+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:25.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:25.149+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:25.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:25.149+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:25.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:25.151+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:25.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:25.829+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:25.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:25.829+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:25.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:25.829+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:25.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:25.829+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:25.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:25.608+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:25.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:25.608+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:25.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:25.608+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:25.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:25.609+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:26.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:26.150+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:26.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:26.150+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:26.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:26.150+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:26.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:26.151+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:26.845 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:13:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:13:26.531+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 11884 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:13:26.845 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:26.624+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:26.845 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:26.624+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:26.845 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:26.624+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:26.845 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:26.625+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:27.130 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:26.841+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:27.131 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:26.841+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:27.131 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:26.841+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:27.131 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:26.841+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:27.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:27.126+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:27.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:27.126+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:27.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:27.126+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:27.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:27.127+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:27.760 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:27.649+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:27.760 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:27.649+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:27.760 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:27.649+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:27.760 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:27.650+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:28.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:28.092+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:28.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:28.092+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:28.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:28.092+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:28.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:28.093+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:28.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:27.888+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:28.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:27.888+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:28.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:27.888+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:28.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:27.890+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:29.091 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:28.846+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:29.091 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:28.846+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:29.091 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:28.846+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:29.091 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:28.847+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:29.092 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:28.686+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:29.092 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:28.686+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:29.092 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:28.686+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:29.092 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:28.687+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:29.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:29.087+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:29.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:29.087+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:29.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:29.087+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:29.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:29.088+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:29.811 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:29.674+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:29.811 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:29.674+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:29.812 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:29.674+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:29.812 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:29.675+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:30.091 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:29.806+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:30.092 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:29.806+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:30.092 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:29.806+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:30.092 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:29.807+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:30.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:30.087+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:30.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:30.087+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:30.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:30.087+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:30.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:30.088+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:30.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:30.852+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:30.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:30.852+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:30.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:30.852+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:30.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:30.853+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:30.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:30.650+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:30.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:30.650+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:30.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:30.650+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:30.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:30.651+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:31.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:31.113+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:31.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:31.113+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:31.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:31.113+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:31.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:31.114+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:31.870 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:13:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:13:31.532+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 11979 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:13:31.870 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:31.683+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:31.870 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:31.683+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:31.870 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:31.683+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:31.870 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:31.684+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:32.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:31.866+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:32.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:31.866+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:32.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:31.866+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:32.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:31.867+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:32.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:32.096+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:32.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:32.096+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:32.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:32.096+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:32.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:32.097+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:33.091 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:32.704+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:33.091 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:32.705+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:33.091 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:32.705+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:33.091 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:32.705+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:33.091 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:32.902+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:33.091 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:32.902+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:33.091 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:32.902+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:33.092 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:32.903+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:33.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:33.087+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:33.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:33.087+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:33.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:33.087+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:33.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:33.087+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:33.866 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:33.715+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:33.866 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:33.715+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:33.866 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:33.715+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:33.866 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:33.716+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:34.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:34.061+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:34.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:34.061+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:34.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:34.061+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:34.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:34.061+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:34.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:33.862+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:34.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:33.862+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:34.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:33.862+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:34.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:33.863+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:35.038 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:34.894+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:35.039 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:34.894+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:35.039 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:34.894+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:35.039 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:34.895+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:35.039 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:34.716+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:35.039 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:34.716+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:35.039 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:34.716+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:35.039 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:34.717+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:35.395 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:35.034+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:35.395 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:35.034+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:35.395 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:35.034+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:35.395 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:35.035+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:36.015 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:35.920+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:36.015 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:35.920+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:36.015 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:35.920+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:36.015 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:35.921+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:36.015 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:35.731+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:36.015 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:35.731+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:36.015 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:35.731+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:36.015 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:35.731+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:36.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:36.010+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:36.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:36.010+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:36.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:36.010+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:36.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:36.011+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:36.902 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:13:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:13:36.533+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 12073 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:13:36.902 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:36.728+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:36.902 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:36.728+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:36.903 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:36.728+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:36.903 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:36.729+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:37.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:36.898+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:37.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:36.898+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:37.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:36.898+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:37.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:36.899+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:37.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:37.004+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:37.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:37.004+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:37.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:37.004+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:37.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:37.005+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:38.017 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:37.872+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:38.017 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:37.872+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:38.017 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:37.872+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:38.017 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:37.874+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:38.017 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:37.704+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:38.017 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:37.704+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:38.018 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:37.704+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:38.018 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:37.705+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:38.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:38.013+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:38.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:38.013+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:38.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:38.013+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:38.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:38.014+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:38.972 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:38.920+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:38.972 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:38.920+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:38.972 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:38.920+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:38.972 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:38.921+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:38.973 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:38.712+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:38.973 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:38.712+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:38.973 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:38.712+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:38.973 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:38.713+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:39.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:38.968+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:39.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:38.968+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:39.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:38.968+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:39.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:38.969+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:40.000 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:39.922+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:40.001 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:39.922+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:40.001 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:39.922+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:40.001 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:39.923+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:40.001 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:39.723+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:40.001 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:39.723+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:40.001 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:39.723+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:40.001 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:39.724+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:40.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:39.996+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:40.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:39.996+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:40.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:39.996+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:40.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:39.997+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:41.026 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:40.918+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:41.026 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:40.918+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:41.026 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:40.918+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:41.026 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:40.919+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:41.026 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:40.712+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:41.026 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:40.712+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:41.026 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:40.712+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:41.026 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:40.712+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:41.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:41.021+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:41.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:41.021+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:41.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:41.021+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:41.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:41.022+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:41.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:13:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:13:41.534+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 12167 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:13:41.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:41.670+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:41.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:41.670+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:41.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:41.670+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:41.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:41.671+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:42.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:41.959+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:42.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:41.959+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:42.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:41.959+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:42.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:41.960+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:42.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:42.006+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:42.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:42.006+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:42.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:42.006+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:42.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:42.007+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:42.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-02-09T22:13:42.684+0000 7f7a97bd5700 0 monclient(hunting): authenticate timed out after 300 2024-02-09T22:13:42.690 INFO:teuthology.orchestra.run.smithi006.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-02-09T22:13:42.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:42.631+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:42.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:42.631+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:42.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:42.631+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:42.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:42.632+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:43.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:42.982+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:43.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:42.982+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:43.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:42.982+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:43.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:42.983+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:42.991+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:42.991+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:42.991+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:42.992+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:43.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T22:13:43.574 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 1402, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 40, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi006 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2024-02-09T22:13:43.792 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=854a4cf6212d44adb2dea634b979eb15 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 1402, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 40, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi006 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2024-02-09T22:13:43.796 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-02-09T22:13:43.849 INFO:tasks.kclient:Unmounting kernel clients... 2024-02-09T22:13:43.850 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-02-09T22:13:43.850 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-09T22:13:43.872 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-02-09T22:13:43.872 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-09T22:13:43.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:43.623+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:43.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:43.623+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:43.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:43.623+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:43.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:43.624+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:43.924 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-09T22:13:44.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:44.001+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:44.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:44.001+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:44.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:44.001+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:44.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:44.002+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:44.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:43.952+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:44.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:43.952+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:44.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:43.952+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:44.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:43.953+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:44.948 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:44.669+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:44.949 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:44.669+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:44.949 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:44.669+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:44.949 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:44.670+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:45.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:45.013+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:45.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:45.013+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:45.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:45.013+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:45.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:45.014+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:45.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:44.945+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:45.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:44.945+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:45.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:44.945+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:45.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:44.946+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:46.044 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:45.698+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:46.044 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:45.698+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:46.044 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:45.698+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:46.044 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:45.699+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:46.045 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:45.942+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:46.045 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:45.942+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:46.045 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:45.942+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:46.045 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:45.943+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:46.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:46.040+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:46.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:46.040+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:46.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:46.040+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:46.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:46.041+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:46.903 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:13:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:13:46.536+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 12260 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:13:46.903 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:46.718+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:46.903 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:46.718+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:46.903 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:46.718+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:46.903 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:46.719+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:47.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:47.032+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:47.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:47.032+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:47.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:47.032+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:47.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:47.033+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:47.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:46.899+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:47.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:46.899+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:47.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:46.899+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:47.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:46.900+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:48.030 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:47.932+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:48.030 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:47.932+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:48.030 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:47.932+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:48.030 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:47.932+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:48.031 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:47.692+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:48.031 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:47.692+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:48.031 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:47.692+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:48.031 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:47.693+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:48.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:48.026+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:48.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:48.026+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:48.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:48.026+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:48.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:48.027+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:48.986 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:48.898+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:48.986 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:48.898+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:48.986 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:48.898+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:48.986 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:48.899+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:48.987 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:48.720+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:48.987 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:48.720+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:48.987 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:48.720+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:48.987 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:48.722+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:49.401 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:48.981+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:49.401 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:48.981+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:49.402 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:48.981+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:49.402 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:48.982+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:50.014 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:49.856+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:50.014 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:49.856+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:50.014 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:49.856+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:50.014 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:49.857+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:50.015 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:49.695+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:50.015 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:49.695+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:50.015 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:49.695+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:50.015 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:49.695+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:50.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:50.010+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:50.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:50.010+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:50.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:50.010+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:50.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:50.011+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:50.981 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:50.892+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:50.981 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:50.892+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:50.981 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:50.892+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:50.981 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:50.892+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:50.981 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:50.694+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:50.982 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:50.694+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:50.982 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:50.694+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:50.982 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:50.695+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:51.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:50.976+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:51.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:50.976+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:51.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:50.976+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:51.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:50.977+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:51.888 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:13:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:13:51.537+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 12355 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:13:51.889 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:51.662+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:51.889 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:51.662+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:51.889 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:51.662+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:51.889 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:51.663+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:52.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:51.977+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:52.160 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:51.977+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:52.160 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:51.977+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:52.160 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:51.977+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:52.160 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:51.884+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:52.160 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:51.884+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:52.161 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:51.884+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:52.161 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:51.885+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:52.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:52.839+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:52.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:52.839+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:52.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:52.839+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:52.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:52.840+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:52.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:52.640+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:52.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:52.640+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:52.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:52.640+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:52.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:52.641+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:53.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:52.971+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:53.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:52.971+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:53.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:52.971+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:53.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:52.971+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:53.820 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:53.664+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:53.820 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:53.664+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:53.820 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:53.664+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:53.820 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:53.664+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:54.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:53.926+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:54.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:53.926+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:54.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:53.926+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:54.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:53.927+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:54.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:53.815+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:54.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:53.815+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:54.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:53.815+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:54.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:53.816+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:54.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:54.831+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:54.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:54.831+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:54.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:54.831+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:54.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:54.832+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:54.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:54.619+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:54.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:54.619+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:54.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:54.619+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:54.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:54.619+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:55.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:54.914+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:55.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:54.914+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:55.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:54.914+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:55.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:54.915+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:56.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:55.860+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:56.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:55.860+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:56.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:55.861+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:56.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:55.861+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:56.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:55.903+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:56.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:55.903+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:56.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:55.903+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:56.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:55.904+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:56.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:55.660+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:56.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:55.660+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:56.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:55.660+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:56.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:55.661+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:56.880 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:13:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:13:56.538+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 12449 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:13:56.881 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:56.668+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:56.881 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:56.668+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:56.881 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:56.668+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:56.882 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:56.669+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:57.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:56.936+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:57.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:56.936+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:57.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:56.936+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:57.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:56.937+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:57.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:56.876+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:57.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:56.876+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:57.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:56.876+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:57.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:56.877+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:58.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:57.879+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:58.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:57.879+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:58.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:57.879+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:58.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:57.880+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:58.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:57.954+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:58.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:57.954+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:58.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:57.954+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:58.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:57.954+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:58.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:57.706+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:58.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:57.706+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:58.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:57.706+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:58.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:57.706+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:58.854 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:58.663+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:58.855 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:58.663+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:58.855 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:58.663+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:58.855 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:58.663+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:13:59.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:58.978+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:59.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:58.978+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:59.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:58.978+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:13:59.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:13:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:13:58.979+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:59.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:58.850+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:59.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:58.850+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:59.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:58.850+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:13:59.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:58.851+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:13:59.883 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:59.619+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:59.884 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:59.619+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:59.884 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:59.620+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:13:59.884 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:13:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:13:59.621+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:00.151 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:00.013+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:00.151 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:00.013+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:00.151 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:00.013+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:00.152 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:00.013+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:00.152 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:59.879+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:00.152 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:59.879+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:00.152 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:59.879+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:00.152 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:13:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:13:59.881+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:01.006 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:00.847+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:01.006 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:00.847+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:01.006 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:00.847+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:01.006 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:00.847+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:01.007 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:00.665+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:01.007 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:00.665+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:01.007 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:00.665+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:01.007 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:00.666+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:01.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:01.002+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:01.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:01.002+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:01.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:01.002+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:01.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:01.002+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:01.822 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:14:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:14:01.540+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 12543 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:14:01.823 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:01.683+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:01.823 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:01.683+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:01.823 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:01.683+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:01.823 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:01.684+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:02.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:01.986+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:02.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:01.986+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:02.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:01.986+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:02.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:01.986+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:02.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:01.818+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:02.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:01.818+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:02.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:01.818+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:02.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:01.820+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:02.942 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:02.680+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:02.943 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:02.680+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:02.943 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:02.680+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:02.943 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:02.682+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:02.943 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:02.793+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:02.943 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:02.793+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:02.944 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:02.793+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:02.944 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:02.794+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:03.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:02.938+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:03.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:02.938+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:03.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:02.938+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:03.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:02.940+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:03.831 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:03.649+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:03.831 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:03.649+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:03.831 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:03.649+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:03.831 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:03.650+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:04.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:03.827+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:04.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:03.827+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:04.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:03.827+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:04.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:03.828+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:04.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:03.936+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:04.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:03.936+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:04.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:03.936+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:04.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:03.937+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:04.877 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:04.621+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:04.877 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:04.621+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:04.878 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:04.621+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:04.878 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:04.622+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:05.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:04.917+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:05.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:04.917+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:05.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:04.917+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:05.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:04.918+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:05.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:04.873+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:05.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:04.873+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:05.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:04.873+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:05.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:04.874+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:05.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:05.589+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:05.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:05.589+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:05.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:05.589+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:05.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:05.589+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:06.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:05.926+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:06.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:05.926+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:06.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:05.926+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:06.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:05.927+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:06.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:05.906+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:06.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:05.906+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:06.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:05.906+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:06.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:05.907+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:06.877 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:14:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:14:06.541+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 12635 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:14:06.878 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:06.601+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:06.878 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:06.601+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:06.878 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:06.601+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:06.878 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:06.602+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:07.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:06.971+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:07.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:06.971+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:07.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:06.971+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:07.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:06.973+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:07.160 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:06.873+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:07.160 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:06.873+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:07.160 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:06.873+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:07.160 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:06.874+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:07.847 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:07.568+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:07.848 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:07.568+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:07.848 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:07.568+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:07.848 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:07.569+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:08.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:07.974+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:08.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:07.974+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:08.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:07.974+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:08.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:07.975+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:08.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:07.843+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:08.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:07.843+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:08.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:07.843+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:08.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:07.844+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:08.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:08.583+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:08.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:08.583+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:08.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:08.583+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:08.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:08.583+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:08.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:08.810+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:08.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:08.810+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:08.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:08.810+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:08.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:08.811+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:09.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:08.994+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:09.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:08.994+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:09.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:08.994+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:09.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:08.995+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:09.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:09.625+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:09.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:09.625+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:09.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:09.625+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:09.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:09.626+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:09.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:09.858+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:09.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:09.858+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:09.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:09.858+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:09.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:09.859+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:10.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:09.997+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:10.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:09.997+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:10.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:09.997+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:10.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:09.998+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:10.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:10.592+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:10.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:10.592+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:10.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:10.592+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:10.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:10.593+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:10.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:10.817+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:10.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:10.817+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:10.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:10.817+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:10.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:10.817+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:11.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:10.987+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:11.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:10.987+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:11.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:10.987+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:11.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:10.988+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:11.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:14:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:14:11.542+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 12726 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:14:11.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:11.790+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:11.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:11.790+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:11.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:11.790+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:11.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:11.791+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:11.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:11.568+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:11.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:11.568+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:11.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:11.568+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:11.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:11.569+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:12.302 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:11.996+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:12.302 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:11.996+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:12.302 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:11.996+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:12.302 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:11.997+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:12.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:12.605+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:12.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:12.605+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:12.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:12.605+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:12.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:12.606+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:12.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:12.822+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:12.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:12.822+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:12.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:12.822+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:12.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:12.823+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:13.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:12.954+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:13.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:12.954+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:13.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:12.954+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:13.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:12.955+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:13.838 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:13.611+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:13.839 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:13.611+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:13.839 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:13.611+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:13.839 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:13.612+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:14.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:13.933+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:14.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:13.933+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:14.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:13.933+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:14.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:13.934+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:14.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:13.835+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:14.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:13.835+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:14.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:13.835+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:14.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:13.835+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:15.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:14.892+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:15.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:14.892+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:15.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:14.892+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:15.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:14.893+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:15.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:14.661+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:15.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:14.661+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:15.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:14.661+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:15.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:14.662+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:15.160 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:14.853+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:15.160 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:14.853+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:15.161 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:14.853+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:15.161 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:14.854+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:15.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:15.647+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:15.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:15.647+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:15.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:15.647+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:15.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:15.648+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:15.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:15.886+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:15.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:15.886+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:15.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:15.886+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:15.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:15.888+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:15.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:15.876+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:15.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:15.876+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:15.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:15.876+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:15.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:15.877+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:16.869 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:14:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:14:16.543+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 12819 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:14:16.869 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:16.690+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:16.869 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:16.690+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:16.869 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:16.690+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:16.869 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:16.691+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:17.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:16.867+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:17.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:16.867+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:17.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:16.867+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:17.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:16.868+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:17.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:16.864+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:17.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:16.864+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:17.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:16.864+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:17.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:16.864+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:18.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:17.826+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:18.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:17.826+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:18.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:17.826+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:18.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:17.826+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:18.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:17.663+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:18.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:17.663+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:18.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:17.663+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:18.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:17.663+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:18.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:17.877+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:18.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:17.877+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:18.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:17.877+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:18.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:17.877+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:18.866 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:18.646+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:18.866 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:18.646+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:18.866 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:18.646+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:18.867 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:18.647+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:19.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:18.858+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:19.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:18.858+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:19.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:18.858+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:19.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:18.860+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:19.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:18.873+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:19.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:18.873+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:19.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:18.873+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:19.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:18.874+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:19.889 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:19.838+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:19.889 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:19.838+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:19.890 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:19.838+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:19.890 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:19.838+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:19.890 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:19.603+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:19.890 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:19.603+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:19.890 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:19.603+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:19.890 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:19.604+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:20.151 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:19.885+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:20.151 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:19.885+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:20.152 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:19.885+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:20.152 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:19.885+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:20.867 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:20.575+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:20.867 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:20.575+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:20.868 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:20.575+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:20.868 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:20.576+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:21.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:20.927+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:21.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:20.927+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:21.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:20.927+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:21.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:20.928+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:21.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:20.863+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:21.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:20.863+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:21.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:20.863+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:21.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:20.863+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:21.883 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:14:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:14:21.545+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 12912 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:14:21.883 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:21.578+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:21.884 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:21.578+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:21.884 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:21.578+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:21.884 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:21.578+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:22.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:21.902+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:22.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:21.902+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:22.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:21.902+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:22.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:21.903+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:22.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:21.879+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:22.160 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:21.879+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:22.160 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:21.879+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:22.160 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:21.880+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:22.874 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:22.529+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:22.874 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:22.529+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:22.874 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:22.529+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:22.875 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:22.530+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:23.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:22.870+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:23.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:22.870+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:23.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:22.870+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:23.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:22.871+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:23.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:22.909+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:23.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:22.909+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:23.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:22.909+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:23.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:22.910+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:23.833 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:23.527+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:23.833 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:23.527+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:23.833 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:23.527+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:23.833 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:23.528+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:24.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:23.828+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:24.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:23.828+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:24.160 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:23.828+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:24.160 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:23.829+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:24.160 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:23.890+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:24.160 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:23.890+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:24.161 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:23.890+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:24.161 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:23.891+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:24.811 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:24.482+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:24.812 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:24.482+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:24.812 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:24.482+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:24.812 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:24.483+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:25.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:24.880+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:25.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:24.880+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:25.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:24.880+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:25.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:24.881+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:25.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:24.807+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:25.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:24.807+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:25.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:24.807+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:25.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:24.808+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:25.792 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:25.518+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:25.792 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:25.518+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:25.792 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:25.518+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:25.793 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:25.519+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:26.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:25.788+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:26.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:25.788+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:26.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:25.788+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:26.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:25.789+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:26.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:25.853+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:26.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:25.853+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:26.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:25.853+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:26.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:25.853+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:26.794 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:14:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:14:26.546+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 13004 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:14:26.795 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:26.509+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:26.795 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:26.509+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:26.795 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:26.509+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:26.795 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:26.510+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:27.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:26.790+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:27.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:26.790+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:27.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:26.790+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:27.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:26.791+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:27.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:26.822+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:27.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:26.822+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:27.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:26.822+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:27.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:26.823+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:27.815 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:27.533+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:27.815 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:27.533+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:27.815 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:27.533+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:27.815 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:27.534+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:28.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:27.811+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:28.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:27.811+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:28.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:27.811+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:28.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:27.812+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:28.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:27.822+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:28.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:27.822+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:28.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:27.822+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:28.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:27.823+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:28.813 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:28.531+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:28.813 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:28.531+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:28.813 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:28.531+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:28.814 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:28.533+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:29.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:28.809+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:29.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:28.809+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:29.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:28.809+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:29.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:28.810+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:29.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:28.865+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:29.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:28.865+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:29.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:28.865+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:29.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:28.866+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:29.816 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:29.489+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:29.817 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:29.489+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:29.817 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:29.489+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:29.817 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:29.489+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:30.152 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:29.873+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:30.152 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:29.873+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:30.152 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:29.874+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:30.152 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:29.875+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:30.152 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:29.812+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:30.153 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:29.812+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:30.153 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:29.812+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:30.153 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:29.814+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:30.734 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:30.448+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:30.735 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:30.448+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:30.735 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:30.448+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:30.735 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:30.449+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:31.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:30.832+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:31.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:30.832+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:31.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:30.832+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:31.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:30.833+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:31.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:30.906+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:31.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:30.906+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:31.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:30.906+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:31.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:30.907+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:31.830 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:14:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:14:31.548+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 13098 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:14:31.831 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:31.453+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:31.831 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:31.453+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:31.831 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:31.453+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:31.831 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:31.454+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:32.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:31.826+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:32.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:31.826+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:32.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:31.826+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:32.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:31.827+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:32.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:31.892+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:32.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:31.893+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:32.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:31.893+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:32.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:31.894+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:32.844 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:32.472+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:32.844 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:32.472+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:32.844 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:32.472+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:32.845 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:32.472+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:33.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:32.858+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:33.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:32.858+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:33.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:32.858+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:33.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:32.859+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:33.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:32.840+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:33.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:32.840+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:33.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:32.840+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:33.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:32.841+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:33.806 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:33.497+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:33.807 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:33.497+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:33.807 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:33.497+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:33.807 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:33.498+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:34.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:33.900+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:34.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:33.900+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:34.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:33.900+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:34.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:33.901+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:34.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:33.869+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:34.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:33.869+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:34.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:33.869+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:34.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:33.870+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:34.858 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:34.521+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:34.859 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:34.521+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:34.859 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:34.521+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:34.859 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:34.521+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:35.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:34.854+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:35.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:34.854+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:35.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:34.854+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:35.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:34.855+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:35.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:34.890+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:35.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:34.890+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:35.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:34.890+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:35.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:34.891+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:35.868 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:35.501+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:35.868 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:35.501+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:35.868 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:35.501+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:35.868 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:35.502+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:36.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:35.864+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:36.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:35.864+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:36.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:35.864+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:36.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:35.865+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:36.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:35.895+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:36.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:35.895+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:36.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:35.895+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:36.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:35.896+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:36.892 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:14:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:14:36.549+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 13192 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:14:36.892 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:36.476+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:36.893 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:36.476+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:36.893 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:36.476+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:36.893 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:36.477+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:37.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:36.929+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:37.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:36.929+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:37.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:36.929+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:37.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:36.930+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:37.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:36.888+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:37.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:36.888+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:37.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:36.888+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:37.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:36.889+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:37.906 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:37.445+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:37.906 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:37.445+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:37.906 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:37.445+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:37.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:37.446+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:38.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:37.926+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:38.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:37.926+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:38.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:37.926+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:38.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:37.926+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:38.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:37.902+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:38.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:37.902+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:38.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:37.902+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:38.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:37.903+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:38.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:38.437+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:38.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:38.437+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:38.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:38.437+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:38.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:38.437+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:39.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:38.976+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:39.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:38.976+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:39.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:38.976+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:39.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:38.977+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:39.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:38.937+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:39.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:38.937+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:39.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:38.937+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:39.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:38.938+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:39.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:39.411+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:39.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:39.411+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:39.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:39.411+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:39.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:39.411+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:40.367 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:39.970+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:40.367 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:39.970+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:40.368 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:39.970+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:40.368 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:39.971+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:40.368 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:39.915+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:40.368 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:39.915+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:40.368 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:39.915+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:40.368 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:39.916+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:40.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:40.363+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:40.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:40.363+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:40.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:40.363+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:40.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:40.364+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:41.397 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:40.940+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:41.397 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:40.940+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:41.397 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:40.940+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:41.397 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:40.941+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:41.398 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:40.971+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:41.398 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:40.971+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:41.398 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:40.971+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:41.398 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:40.972+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:41.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:14:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:14:41.551+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 13285 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:14:41.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:41.393+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:41.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:41.393+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:41.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:41.393+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:41.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:41.394+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:42.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:41.953+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:42.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:41.953+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:42.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:41.953+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:42.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:41.954+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:42.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:41.955+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:42.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:41.955+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:42.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:41.955+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:42.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:41.956+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:42.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:42.408+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:42.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:42.408+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:42.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:42.408+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:42.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:42.409+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:43.394 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:42.948+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:43.394 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:42.948+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:43.394 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:42.948+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:43.395 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:42.949+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:43.395 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:42.936+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:43.395 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:42.936+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:43.395 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:42.936+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:43.395 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:42.936+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:43.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:43.390+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:43.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:43.390+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:43.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:43.390+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:43.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:43.391+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:44.359 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:43.985+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:44.359 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:43.985+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:44.359 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:43.985+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:44.359 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:43.986+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:44.360 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:43.935+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:44.360 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:43.935+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:44.360 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:43.935+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:44.360 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:43.936+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:44.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:44.355+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:44.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:44.355+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:44.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:44.355+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:44.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:44.356+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:45.332 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:44.945+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:45.332 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:44.945+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:45.333 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:44.945+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:45.333 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:44.946+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:45.333 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:44.951+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:45.333 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:44.951+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:45.333 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:44.951+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:45.333 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:44.951+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:45.334 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:45.328+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:45.334 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:45.328+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:45.334 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:45.328+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:45.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:45.329+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:46.311 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:45.947+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:46.311 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:45.947+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:46.311 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:45.947+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:46.311 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:45.948+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:46.311 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:45.995+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:46.312 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:45.995+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:46.312 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:45.995+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:46.312 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:45.996+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:46.569 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:14:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:14:46.552+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 13378 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:14:46.570 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:46.307+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:46.570 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:46.307+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:46.570 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:46.307+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:46.570 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:46.308+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:47.334 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:46.997+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:47.334 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:46.997+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:47.334 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:46.997+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:47.334 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:46.997+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:47.334 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:46.931+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:47.335 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:46.931+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:47.335 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:46.931+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:47.335 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:46.932+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:47.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:47.329+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:47.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:47.329+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:47.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:47.329+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:47.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:47.331+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:48.325 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:47.931+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:48.325 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:47.931+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:48.325 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:47.931+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:48.325 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:47.931+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:48.325 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:47.968+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:48.326 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:47.968+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:48.326 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:47.968+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:48.326 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:47.969+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:48.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:48.321+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:48.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:48.321+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:48.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:48.321+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:48.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:48.321+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:49.306 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:48.926+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:49.307 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:48.926+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:49.307 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:48.926+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:49.307 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:48.926+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:49.307 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:49.014+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:49.307 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:49.014+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:49.307 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:49.014+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:49.308 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:49.014+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:49.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:49.302+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:49.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:49.302+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:49.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:49.302+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:49.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:49.304+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:50.348 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:49.953+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:50.349 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:49.953+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:50.349 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:49.953+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:50.349 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:49.954+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:50.349 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:50.012+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:50.349 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:50.012+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:50.350 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:50.012+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:50.350 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:50.013+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:50.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:50.344+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:50.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:50.344+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:50.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:50.344+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:50.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:50.345+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:51.317 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:50.905+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:51.317 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:50.905+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:51.317 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:50.905+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:51.317 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:50.906+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:51.318 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:50.999+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:51.318 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:50.999+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:51.318 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:50.999+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:51.318 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:51.000+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:51.570 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:14:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:14:51.553+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 13472 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:14:51.570 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:51.313+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:51.571 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:51.313+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:51.571 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:51.313+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:51.571 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:51.314+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:52.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:52.037+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:52.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:52.037+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:52.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:52.037+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:52.160 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:52.037+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:52.160 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:51.902+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:52.160 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:51.902+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:52.160 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:51.902+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:52.161 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:51.903+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:52.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:52.280+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:52.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:52.280+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:52.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:52.280+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:52.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:52.281+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:53.273 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:52.952+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:53.273 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:52.952+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:53.273 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:52.952+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:53.273 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:52.953+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:53.273 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:53.044+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:53.274 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:53.044+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:53.274 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:53.044+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:53.274 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:53.045+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:53.580 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:53.269+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:53.580 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:53.269+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:53.580 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:53.269+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:53.580 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:53.269+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:54.269 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:53.956+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:54.270 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:53.956+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:54.270 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:53.956+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:54.270 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:53.957+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:54.270 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:54.039+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:54.270 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:54.039+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:54.271 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:54.039+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:54.271 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:54.040+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:54.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:54.265+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:54.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:54.265+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:54.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:54.265+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:54.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:54.266+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:55.310 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:54.989+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:55.310 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:54.989+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:55.310 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:54.989+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:55.311 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:54.991+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:55.311 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:55.038+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:55.311 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:55.038+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:55.311 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:55.038+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:55.311 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:55.039+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:55.312 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:55.236+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:55.312 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:55.236+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:55.312 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:55.236+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:55.312 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:55.237+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:56.228 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:55.947+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:56.229 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:55.947+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:56.229 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:55.947+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:56.229 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:55.948+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:56.229 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:56.017+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:56.229 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:56.017+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:56.230 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:56.017+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:56.230 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:56.018+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:56.559 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:56.225+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:56.559 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:56.225+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:56.559 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:56.225+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:56.559 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:56.226+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:56.905 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:14:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:14:56.555+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 13566 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:14:57.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:56.901+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:57.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:56.901+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:57.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:56.901+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:57.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:56.902+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:57.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:56.984+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:57.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:56.984+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:57.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:56.984+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:57.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:56.985+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:57.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:57.220+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:57.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:57.220+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:57.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:57.220+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:57.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:57.222+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:58.257 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:57.963+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:58.257 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:57.963+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:58.257 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:57.963+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:58.257 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:57.964+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:58.258 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:57.923+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:58.258 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:57.923+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:58.258 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:57.923+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:58.258 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:57.923+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:58.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:58.253+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:58.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:58.253+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:58.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:58.253+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:58.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:58.253+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:14:59.272 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:58.955+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:59.272 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:58.955+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:59.273 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:58.955+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:14:59.273 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:58.957+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:59.273 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:58.916+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:59.273 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:58.916+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:59.273 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:58.916+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:14:59.273 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:58.917+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:14:59.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:59.268+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:59.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:59.268+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:59.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:59.268+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:14:59.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:14:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:14:59.269+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:00.236 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:59.935+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:00.236 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:59.935+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:00.237 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:59.935+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:00.237 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:14:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:14:59.936+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:00.237 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:59.937+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:00.237 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:59.937+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:00.237 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:59.937+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:00.237 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:14:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:14:59.939+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:00.238 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:00.232+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:00.238 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:00.232+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:00.238 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:00.232+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:00.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:00.233+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:01.287 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:00.918+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:01.287 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:00.918+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:01.287 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:00.918+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:01.287 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:00.919+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:01.288 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:00.956+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:01.288 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:00.956+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:01.288 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:00.956+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:01.288 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:00.957+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:01.560 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:01.283+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:01.560 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:01.283+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:01.561 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:01.283+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:01.561 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:01.284+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:01.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:15:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:15:01.556+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 13660 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:15:02.247 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:01.931+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:02.247 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:01.931+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:02.247 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:01.931+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:02.247 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:01.932+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:02.247 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:01.912+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:02.248 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:01.912+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:02.248 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:01.912+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:02.248 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:01.913+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:02.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:02.242+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:02.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:02.242+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:02.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:02.242+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:02.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:02.243+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:03.198 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:02.978+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:03.198 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:02.978+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:03.198 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:02.978+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:03.198 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:02.979+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:03.198 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:02.918+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:03.198 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:02.918+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:03.199 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:02.918+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:03.199 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:02.919+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:03.498 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:03.194+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:03.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:03.194+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:03.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:03.194+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:03.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:03.195+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:04.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:03.937+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:04.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:03.937+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:04.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:03.937+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:04.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:03.937+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:03.939+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:03.939+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:04.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:03.939+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:04.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:03.939+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:04.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:04.154+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:04.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:04.154+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:04.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:04.154+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:04.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:04.155+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:05.175 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:04.932+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:05.175 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:04.932+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:05.175 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:04.932+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:05.175 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:04.933+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:05.176 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:04.905+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:05.176 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:04.905+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:05.176 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:04.905+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:05.176 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:04.906+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:05.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:05.170+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:05.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:05.170+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:05.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:05.171+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:05.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:05.172+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:06.128 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:05.884+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:06.128 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:05.884+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:06.129 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:05.884+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:06.129 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:05.885+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:06.129 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:05.866+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:06.129 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:05.867+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:06.129 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:05.867+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:06.129 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:05.868+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:06.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:06.124+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:06.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:06.124+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:06.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:06.124+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:06.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:06.125+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:06.895 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:15:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:15:06.558+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 13753 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:15:07.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:06.916+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:07.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:06.916+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:07.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:06.916+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:07.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:06.917+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:07.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:07.107+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:07.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:07.108+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:07.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:07.108+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:07.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:07.108+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:07.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:06.891+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:07.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:06.891+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:07.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:06.891+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:07.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:06.892+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:08.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:07.941+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:08.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:07.941+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:08.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:07.941+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:08.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:07.942+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:08.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:07.877+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:08.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:07.877+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:08.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:07.877+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:08.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:07.878+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:08.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:08.155+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:08.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:08.155+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:08.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:08.155+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:08.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:08.156+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:09.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:08.902+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:09.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:08.902+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:09.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:08.902+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:09.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:08.903+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:09.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:09.115+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:09.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:09.115+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:09.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:09.115+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:09.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:09.116+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:09.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:08.929+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:09.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:08.929+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:09.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:08.929+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:09.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:08.930+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:10.139 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:09.879+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:10.139 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:09.879+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:10.140 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:09.879+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:10.140 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:09.880+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:10.140 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:09.901+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:10.141 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:09.901+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:10.141 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:09.901+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:10.141 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:09.902+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:10.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:10.135+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:10.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:10.135+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:10.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:10.135+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:10.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:10.136+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:11.113 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:10.905+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:11.113 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:10.905+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:11.113 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:10.905+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:11.113 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:10.907+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:11.113 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:10.866+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:11.114 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:10.866+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:11.114 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:10.866+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:11.114 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:10.867+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:11.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:11.109+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:11.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:11.109+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:11.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:11.109+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:11.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:11.110+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:11.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:15:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:15:11.559+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 13848 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:15:12.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:11.937+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:12.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:11.937+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:12.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:11.937+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:12.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:11.938+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:12.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:11.904+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:12.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:11.904+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:12.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:11.904+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:12.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:11.906+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:12.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:12.088+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:12.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:12.088+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:12.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:12.088+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:12.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:12.089+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:13.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:12.943+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:13.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:12.943+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:13.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:12.943+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:13.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:12.944+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:13.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:13.040+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:13.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:13.040+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:13.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:13.040+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:13.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:13.041+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:13.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:12.899+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:13.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:12.899+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:13.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:12.899+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:13.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:12.900+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:14.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:13.906+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:14.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:13.906+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:14.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:13.906+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:14.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:13.906+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:14.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:14.004+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:14.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:14.004+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:14.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:14.004+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:14.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:14.005+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:14.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:13.918+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:14.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:13.918+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:14.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:13.918+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:14.411 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:13.918+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:15.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:14.943+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:15.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:14.943+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:15.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:14.943+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:15.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:14.944+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:15.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:14.941+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:15.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:14.941+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:15.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:14.941+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:15.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:14.942+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:15.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:15.034+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:15.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:15.034+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:15.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:15.034+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:15.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:15.034+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:16.060 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:15.975+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:16.061 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:15.975+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:16.061 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:15.975+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:16.061 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:15.976+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:16.061 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:15.925+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:16.061 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:15.925+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:16.061 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:15.925+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:16.062 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:15.926+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:16.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:16.056+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:16.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:16.056+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:16.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:16.056+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:16.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:16.057+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:16.814 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:15:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:15:16.561+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 13943 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:15:17.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:16.975+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:17.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:16.975+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:17.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:16.975+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:17.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:16.976+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:17.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:16.900+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:17.160 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:16.900+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:17.160 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:16.900+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:17.160 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:16.900+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:17.161 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:17.091+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:17.161 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:17.091+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:17.161 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:17.091+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:17.162 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:17.092+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:18.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:17.940+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:18.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:17.940+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:18.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:17.940+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:18.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:17.941+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:18.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:17.900+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:18.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:17.900+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:18.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:17.900+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:18.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:17.901+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:18.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:18.058+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:18.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:18.058+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:18.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:18.058+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:18.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:18.059+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:19.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:18.868+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:19.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:18.868+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:19.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:18.868+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:19.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:18.869+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:19.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:19.071+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:19.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:19.071+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:19.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:19.071+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:19.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:19.072+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:19.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:18.981+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:19.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:18.981+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:19.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:18.981+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:19.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:18.982+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:20.156 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:19.946+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:20.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:19.946+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:20.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:19.946+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:20.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:19.947+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:20.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:20.087+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:20.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:20.087+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:20.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:20.087+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:20.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:20.087+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:20.161 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:19.850+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:20.161 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:19.850+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:20.161 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:19.850+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:20.161 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:19.851+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:21.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:20.975+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:21.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:20.975+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:21.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:20.975+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:21.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:20.976+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:21.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:21.089+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:21.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:21.089+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:21.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:21.089+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:21.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:21.090+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:21.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:20.895+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:21.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:20.895+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:21.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:20.895+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:21.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:20.896+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:21.876 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:15:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:15:21.562+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 14037 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:15:22.129 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:21.872+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:22.130 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:21.872+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:22.130 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:21.872+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:22.130 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:21.873+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:22.130 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:21.951+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:22.130 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:21.951+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:22.131 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:21.951+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:22.131 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:21.952+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:22.125+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:22.125+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:22.125+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:22.127+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:23.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:22.935+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:23.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:22.935+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:23.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:22.935+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:23.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:22.936+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:23.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:23.112+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:23.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:23.112+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:23.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:23.112+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:23.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:23.114+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:23.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:22.892+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:23.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:22.892+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:23.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:22.892+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:23.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:22.892+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:24.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:23.974+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:24.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:23.974+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:24.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:23.974+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:24.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:23.975+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:24.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:23.878+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:24.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:23.878+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:24.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:23.878+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:24.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:23.878+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:24.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:24.162+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:24.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:24.162+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:24.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:24.162+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:24.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:24.163+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:25.153 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:24.946+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:25.153 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:24.946+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:25.153 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:24.946+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:25.154 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:24.947+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:25.154 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:24.875+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:25.154 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:24.875+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:25.154 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:24.875+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:25.154 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:24.876+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:25.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:25.149+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:25.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:25.149+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:25.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:25.149+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:25.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:25.150+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:26.161 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:25.981+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:26.161 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:25.981+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:26.161 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:25.981+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:26.161 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:25.982+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:26.161 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:25.906+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:26.162 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:25.906+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:26.162 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:25.906+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:26.162 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:25.907+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:26.568 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:26.157+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:26.568 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:26.157+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:26.568 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:26.157+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:26.568 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:26.157+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:26.898 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:15:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:15:26.564+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 14126 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:15:27.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:26.894+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:27.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:26.894+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:27.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:26.894+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:27.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:26.895+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:27.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:27.014+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:27.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:27.014+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:27.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:27.014+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:27.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:27.015+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:27.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:27.130+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:27.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:27.130+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:27.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:27.130+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:27.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:27.130+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:28.175 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:28.170+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:28.175 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:28.170+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:28.175 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:28.170+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:28.176 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:28.016+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:28.176 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:28.016+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:28.176 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:28.016+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:28.176 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:28.017+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:28.176 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:27.921+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:28.176 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:27.921+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:28.177 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:27.921+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:28.177 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:27.921+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:28.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:28.172+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:29.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:29.055+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:29.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:29.055+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:29.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:29.055+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:29.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:29.057+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:29.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:29.178+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:29.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:29.178+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:29.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:29.178+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:29.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:29.179+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:29.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:28.939+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:29.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:28.939+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:29.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:28.939+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:29.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:28.940+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:30.156 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:30.013+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:30.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:30.013+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:30.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:30.013+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:30.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:30.014+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:30.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:29.894+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:30.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:29.894+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:30.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:29.894+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:30.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:29.895+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:30.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:30.209+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:30.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:30.209+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:30.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:30.209+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:30.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:30.210+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:31.179 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:31.050+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:31.179 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:31.050+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:31.179 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:31.050+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:31.179 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:31.051+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:31.180 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:30.906+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:31.180 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:30.906+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:31.180 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:30.906+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:31.180 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:30.906+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:31.569 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:31.175+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:31.569 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:31.175+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:31.570 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:31.175+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:31.570 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:31.176+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:31.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:15:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:15:31.565+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 14210 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:15:32.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:32.059+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:32.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:32.059+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:32.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:32.059+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:32.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:32.059+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:32.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:31.951+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:32.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:31.951+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:32.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:31.951+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:32.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:31.952+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:32.411 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:32.162+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:32.411 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:32.162+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:32.411 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:32.162+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:32.411 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:32.163+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:33.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:32.980+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:33.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:32.980+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:33.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:32.980+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:33.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:32.981+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:33.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:33.015+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:33.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:33.015+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:33.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:33.015+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:33.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:33.016+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:33.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:33.159+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:33.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:33.159+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:33.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:33.159+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:33.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:33.159+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:34.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:34.054+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:34.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:34.054+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:34.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:34.054+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:34.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:34.055+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:34.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:33.989+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:34.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:33.989+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:34.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:33.989+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:34.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:33.990+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:34.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:34.173+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:34.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:34.173+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:34.410 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:34.173+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:34.410 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:34.174+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:35.246 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:35.074+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:35.246 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:35.074+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:35.246 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:35.074+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:35.246 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:35.075+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:35.247 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:35.131+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:35.247 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:35.131+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:35.247 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:35.131+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:35.247 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:35.132+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:35.247 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:34.988+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:35.247 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:34.988+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:35.248 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:34.988+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:35.248 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:34.989+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:36.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:36.074+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:36.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:36.074+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:36.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:36.074+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:36.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:36.075+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:36.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:35.998+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:36.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:35.998+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:36.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:35.998+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:36.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:35.998+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:36.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:36.171+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:36.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:36.171+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:36.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:36.171+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:36.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:36.171+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:36.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:15:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:15:36.566+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 14292 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:15:37.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:37.078+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:37.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:37.078+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:37.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:37.078+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:37.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:37.078+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:37.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:37.221+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:37.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:37.221+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:37.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:37.221+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:37.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:37.222+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:37.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:37.029+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:37.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:37.029+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:37.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:37.029+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:37.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:37.030+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:38.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:38.031+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:38.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:38.031+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:38.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:38.031+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:38.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:38.032+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:38.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:38.115+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:38.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:38.115+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:38.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:38.115+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:38.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:38.116+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:38.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:38.205+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:38.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:38.205+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:38.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:38.205+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:38.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:38.205+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:39.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:39.027+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:39.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:39.027+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:39.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:39.027+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:39.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:39.028+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:39.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:39.121+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:39.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:39.121+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:39.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:39.121+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:39.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:39.122+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:39.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:39.220+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:39.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:39.220+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:39.410 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:39.220+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:39.410 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:39.221+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:40.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:40.139+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:40.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:40.139+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:40.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:40.139+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:40.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:40.140+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:40.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:40.031+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:40.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:40.031+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:40.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:40.031+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:40.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:40.031+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:40.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:40.256+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:40.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:40.256+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:40.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:40.256+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:40.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:40.257+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:41.390 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:41.177+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:41.390 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:41.177+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:41.390 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:41.177+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:41.390 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:41.178+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:41.391 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:41.292+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:41.391 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:41.292+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:41.391 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:41.292+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:41.391 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:41.292+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:41.391 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:41.062+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:41.392 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:41.062+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:41.392 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:41.062+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:41.392 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:41.063+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:41.659 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:15:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:15:41.568+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 14375 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:15:42.329 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:42.146+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:42.329 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:42.146+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:42.329 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:42.146+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:42.330 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:42.147+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:42.330 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:42.022+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:42.330 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:42.022+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:42.330 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:42.022+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:42.330 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:42.023+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:42.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:42.325+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:42.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:42.325+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:42.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:42.325+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:42.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:42.326+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:43.338 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:43.194+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:43.339 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:43.194+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:43.339 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:43.194+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:43.339 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:43.194+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:43.339 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:43.047+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:43.339 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:43.047+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:43.339 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:43.047+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:43.340 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:43.048+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:43.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:43.334+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:43.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:43.334+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:43.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:43.334+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:43.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:43.335+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:43.999 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-02-09T22:15:44.000 WARNING:tasks.kclient:Ordinary umount failed, forcing... 2024-02-09T22:15:44.001 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-02-09T22:15:44.001 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-09T22:15:44.019 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-02-09T22:15:44.019 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-09T22:15:44.076 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-09T22:15:44.384 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:44.013+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:44.384 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:44.013+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:44.384 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:44.013+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:44.385 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:44.014+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:44.385 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:44.174+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:44.385 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:44.174+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:44.385 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:44.174+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:44.385 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:44.176+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:44.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:44.380+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:44.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:44.380+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:44.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:44.380+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:44.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:44.381+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:45.356 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:45.179+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:45.356 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:45.179+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:45.356 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:45.179+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:45.356 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:45.180+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:45.357 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:45.026+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:45.357 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:45.026+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:45.357 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:45.026+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:45.357 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:45.026+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:45.357 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:45.352+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:45.357 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:45.352+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:45.358 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:45.352+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:45.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:45.352+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:46.390 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:46.135+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:46.391 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:46.135+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:46.391 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:46.135+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:46.391 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:46.136+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:46.392 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:46.074+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:46.392 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:46.074+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:46.392 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:46.074+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:46.393 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:46.075+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:46.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:15:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:15:46.569+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 14456 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:15:46.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:46.386+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:46.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:46.386+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:46.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:46.386+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:46.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:46.387+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:47.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:47.077+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:47.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:47.077+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:47.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:47.077+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:47.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:47.077+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:47.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:47.104+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:47.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:47.104+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:47.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:47.104+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:47.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:47.105+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:47.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:47.427+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:47.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:47.427+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:47.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:47.427+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:47.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:47.428+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:48.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:48.071+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:48.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:48.071+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:48.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:48.071+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:48.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:48.072+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:48.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:48.106+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:48.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:48.106+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:48.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:48.106+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:48.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:48.107+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:48.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:48.446+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:48.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:48.446+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:48.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:48.446+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:48.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:48.447+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:49.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:49.040+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:49.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:49.040+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:49.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:49.040+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:49.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:49.041+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:49.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:49.063+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:49.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:49.063+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:49.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:49.063+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:49.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:49.064+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:49.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:49.451+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:49.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:49.451+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:49.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:49.451+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:49.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:49.452+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:50.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:50.033+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:50.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:50.033+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:50.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:50.033+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:50.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:50.034+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:50.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:50.084+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:50.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:50.084+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:50.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:50.084+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:50.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:50.085+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:50.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:50.449+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:50.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:50.449+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:50.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:50.449+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:50.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:50.450+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:51.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:51.019+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:51.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:51.019+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:51.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:51.019+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:51.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:51.020+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:51.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:51.124+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:51.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:51.124+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:51.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:51.124+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:51.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:51.125+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:51.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:15:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:15:51.570+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 14540 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:15:51.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:51.451+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:51.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:51.451+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:51.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:51.451+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:51.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:51.452+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:52.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:52.068+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:52.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:52.068+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:52.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:52.068+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:52.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:52.068+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:52.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:52.159+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:52.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:52.159+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:52.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:52.159+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:52.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:52.160+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:52.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:52.480+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:52.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:52.480+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:52.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:52.480+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:52.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:52.481+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:53.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:53.027+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:53.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:53.027+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:53.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:53.027+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:53.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:53.028+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:53.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:53.138+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:53.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:53.138+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:53.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:53.138+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:53.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:53.138+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:53.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:53.513+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:53.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:53.513+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:53.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:53.513+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:53.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:53.513+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:54.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:54.025+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:54.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:54.025+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:54.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:54.025+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:54.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:54.026+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:54.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:54.186+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:54.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:54.186+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:54.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:54.186+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:54.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:54.187+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:54.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:54.517+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:54.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:54.517+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:54.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:54.517+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:54.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:54.518+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:55.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:55.167+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:55.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:55.167+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:55.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:55.167+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:55.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:55.168+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:55.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:55.005+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:55.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:55.005+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:55.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:55.005+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:55.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:55.006+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:55.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:55.554+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:55.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:55.554+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:55.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:55.554+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:55.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:55.555+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:56.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:56.016+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:56.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:56.016+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:56.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:56.016+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:56.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:56.017+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:56.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:56.188+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:56.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:56.188+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:56.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:56.188+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:56.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:56.189+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:56.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:15:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:15:56.571+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 14621 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:15:56.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:56.597+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:56.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:56.597+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:56.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:56.597+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:56.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:56.598+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:57.187 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:57.063+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:57.187 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:57.063+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:57.187 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:57.063+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:57.187 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:57.063+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:57.591 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:57.183+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:57.892 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:57.183+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:57.892 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:57.183+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:57.893 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:57.183+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:57.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:57.586+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:57.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:57.586+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:57.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:57.586+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:57.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:57.586+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:58.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:58.095+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:58.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:58.095+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:58.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:58.095+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:58.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:58.096+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:58.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:58.134+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:58.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:58.134+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:58.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:58.134+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:58.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:58.135+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:58.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:58.554+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:58.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:58.554+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:58.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:58.554+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:58.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:58.555+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:15:59.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:59.141+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:59.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:59.141+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:59.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:59.141+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:15:59.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:15:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:15:59.141+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:59.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:59.058+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:59.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:59.058+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:59.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:59.058+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:15:59.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:15:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:15:59.059+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:15:59.822 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:59.546+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:59.823 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:59.546+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:59.823 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:59.546+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:15:59.823 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:15:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:15:59.547+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:00.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:00.080+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:00.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:00.080+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:00.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:00.080+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:00.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:00.081+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:00.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:00.141+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:00.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:00.141+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:00.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:00.141+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:00.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:00.141+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:00.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:00.575+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:00.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:00.575+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:00.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:00.575+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:00.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:00.576+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:01.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:01.123+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:01.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:01.123+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:01.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:01.123+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:01.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:01.125+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:01.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:01.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:01.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:01.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:01.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:01.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:01.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:01.171+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:01.871 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:16:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:16:01.574+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 14705 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:16:01.871 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:01.544+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:01.871 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:01.544+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:01.871 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:01.544+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:01.871 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:01.545+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:02.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:02.134+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:02.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:02.134+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:02.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:02.134+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:02.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:02.135+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:02.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:02.157+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:02.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:02.157+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:02.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:02.157+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:02.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:02.158+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:02.894 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:02.535+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:02.894 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:02.535+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:02.894 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:02.535+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:02.894 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:02.536+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:03.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:03.123+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:03.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:03.123+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:03.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:03.123+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:03.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:03.123+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:03.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:03.164+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:03.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:03.164+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:03.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:03.164+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:03.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:03.165+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:03.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:03.506+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:03.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:03.506+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:03.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:03.506+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:03.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:03.507+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:04.524 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:04.159+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:04.525 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:04.159+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:04.525 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:04.159+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:04.525 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:04.160+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:04.526 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:04.207+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:04.526 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:04.207+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:04.526 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:04.207+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:04.526 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:04.208+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:04.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:04.520+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:04.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:04.520+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:04.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:04.520+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:04.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:04.521+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:05.485 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:05.169+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:05.485 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:05.169+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:05.485 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:05.169+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:05.485 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:05.170+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:05.485 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:05.157+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:05.486 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:05.157+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:05.486 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:05.157+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:05.486 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:05.158+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:05.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:05.481+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:05.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:05.481+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:05.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:05.481+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:05.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:05.481+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:06.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:06.143+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:06.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:06.143+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:06.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:06.143+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:06.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:06.144+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:06.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:06.122+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:06.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:06.122+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:06.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:06.122+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:06.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:06.123+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:06.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:16:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:16:06.576+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 14786 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:16:06.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:06.493+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:06.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:06.493+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:06.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:06.493+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:06.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:06.493+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:07.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:07.080+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:07.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:07.080+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:07.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:07.080+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:07.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:07.081+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:07.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:07.134+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:07.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:07.134+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:07.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:07.134+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:07.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:07.135+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:07.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:07.493+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:07.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:07.493+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:07.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:07.493+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:07.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:07.493+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:08.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:08.124+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:08.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:08.124+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:08.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:08.125+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:08.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:08.125+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:08.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:08.058+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:08.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:08.058+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:08.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:08.058+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:08.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:08.059+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:08.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:08.539+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:08.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:08.539+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:08.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:08.539+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:08.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:08.540+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:09.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:09.097+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:09.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:09.097+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:09.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:09.097+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:09.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:09.098+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:09.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:09.084+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:09.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:09.084+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:09.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:09.084+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:09.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:09.085+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:09.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:09.528+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:09.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:09.528+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:09.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:09.528+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:09.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:09.529+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:10.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:10.133+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:10.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:10.133+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:10.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:10.133+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:10.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:10.134+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:10.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:10.087+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:10.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:10.087+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:10.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:10.087+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:10.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:10.088+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:10.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:10.548+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:10.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:10.549+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:10.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:10.549+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:10.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:10.550+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:11.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:11.092+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:11.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:11.092+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:11.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:11.092+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:11.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:11.093+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:11.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:11.066+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:11.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:11.066+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:11.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:11.066+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:11.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:11.066+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:11.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:16:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:16:11.577+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 14870 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:16:11.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:11.596+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:11.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:11.596+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:11.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:11.596+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:11.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:11.597+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:12.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:12.081+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:12.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:12.081+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:12.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:12.081+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:12.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:12.082+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:12.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:12.052+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:12.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:12.052+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:12.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:12.052+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:12.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:12.052+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:12.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:12.612+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:12.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:12.612+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:12.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:12.612+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:12.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:12.613+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:13.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:13.111+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:13.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:13.111+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:13.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:13.111+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:13.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:13.112+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:13.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:13.045+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:13.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:13.045+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:13.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:13.045+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:13.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:13.046+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:13.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:13.570+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:13.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:13.570+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:13.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:13.570+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:13.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:13.571+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:14.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:14.074+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:14.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:14.074+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:14.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:14.074+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:14.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:14.074+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:14.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:14.089+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:14.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:14.089+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:14.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:14.089+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:14.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:14.090+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:14.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:14.556+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:14.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:14.556+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:14.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:14.556+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:14.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:14.557+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:15.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:15.124+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:15.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:15.124+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:15.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:15.124+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:15.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:15.125+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:15.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:15.105+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:15.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:15.105+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:15.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:15.105+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:15.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:15.106+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:15.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:15.600+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:15.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:15.600+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:15.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:15.600+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:15.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:15.601+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:16.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:16.074+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:16.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:16.074+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:16.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:16.074+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:16.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:16.074+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:16.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:16.140+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:16.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:16.140+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:16.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:16.140+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:16.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:16.141+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:16.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:16:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:16:16.579+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 14951 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:16:16.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:16.637+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:16.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:16.637+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:16.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:16.637+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:16.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:16.638+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:17.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:17.072+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:17.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:17.072+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:17.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:17.072+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:17.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:17.072+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:17.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:17.179+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:17.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:17.179+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:17.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:17.179+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:17.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:17.180+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:17.742 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:17.635+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:17.743 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:17.635+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:17.743 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:17.635+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:17.743 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:17.635+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:18.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:18.052+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:18.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:18.052+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:18.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:18.052+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:18.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:18.053+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:18.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:18.137+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:18.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:18.137+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:18.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:18.137+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:18.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:18.138+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:18.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:18.646+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:18.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:18.646+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:18.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:18.646+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:18.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:18.647+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:19.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:19.162+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:19.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:19.162+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:19.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:19.162+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:19.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:19.163+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:19.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:19.026+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:19.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:19.026+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:19.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:19.026+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:19.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:19.027+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:19.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:19.607+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:19.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:19.607+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:19.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:19.607+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:19.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:19.608+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:20.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:19.988+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:20.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:19.988+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:20.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:19.988+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:20.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:19.989+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:20.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:20.144+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:20.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:20.144+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:20.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:20.144+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:20.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:20.145+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:20.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:20.559+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:20.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:20.559+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:20.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:20.559+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:20.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:20.560+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:21.326 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:21.170+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:21.326 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:21.170+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:21.326 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:21.170+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:21.326 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:21.171+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:21.327 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:21.023+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:21.327 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:21.023+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:21.327 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:21.023+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:21.327 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:21.024+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:21.584 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:21.540+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:21.584 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:21.540+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:21.585 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:21.540+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:21.585 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:21.540+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:21.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:16:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:16:21.580+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 15035 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:16:22.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:22.169+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:22.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:22.169+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:22.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:22.169+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:22.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:22.170+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:22.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:21.990+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:22.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:21.990+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:22.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:21.990+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:22.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:21.991+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:22.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:22.516+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:22.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:22.516+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:22.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:22.516+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:22.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:22.517+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:23.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:23.035+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:23.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:23.035+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:23.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:23.035+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:23.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:23.035+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:23.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:23.128+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:23.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:23.128+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:23.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:23.128+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:23.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:23.128+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:23.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:23.469+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:23.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:23.469+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:23.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:23.469+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:23.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:23.470+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:24.398 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:24.113+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:24.399 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:24.113+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:24.399 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:24.113+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:24.399 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:24.114+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:24.400 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:24.012+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:24.400 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:24.012+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:24.400 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:24.012+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:24.401 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:24.012+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:24.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:24.500+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:24.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:24.500+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:24.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:24.500+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:24.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:24.501+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:25.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:25.128+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:25.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:25.128+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:25.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:25.128+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:25.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:25.129+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:25.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:25.030+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:25.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:25.030+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:25.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:25.030+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:25.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:25.030+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:25.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:25.526+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:25.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:25.526+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:25.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:25.526+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:25.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:25.526+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:26.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:26.147+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:26.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:26.147+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:26.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:26.147+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:26.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:26.148+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:26.040+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:26.040+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:26.040+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:26.040+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:26.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:16:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:16:26.582+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 15116 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:16:26.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:26.557+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:26.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:26.557+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:26.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:26.557+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:26.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:26.558+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:27.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:27.194+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:27.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:27.194+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:27.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:27.194+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:27.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:27.194+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:27.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:27.045+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:27.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:27.045+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:27.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:27.045+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:27.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:27.046+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:27.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:27.529+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:27.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:27.529+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:27.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:27.529+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:27.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:27.530+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:28.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:28.081+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:28.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:28.081+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:28.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:28.081+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:28.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:28.082+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:28.520 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:28.154+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:28.520 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:28.154+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:28.521 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:28.154+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:28.521 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:28.155+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:28.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:28.516+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:28.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:28.516+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:28.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:28.516+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:28.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:28.517+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:29.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:29.095+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:29.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:29.095+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:29.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:29.095+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:29.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:29.096+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:29.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:29.203+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:29.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:29.203+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:29.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:29.203+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:29.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:29.204+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:29.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:29.477+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:29.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:29.477+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:29.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:29.477+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:29.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:29.478+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:30.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:30.121+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:30.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:30.121+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:30.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:30.121+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:30.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:30.122+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:30.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:30.185+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:30.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:30.185+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:30.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:30.185+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:30.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:30.186+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:30.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:30.435+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:30.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:30.435+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:30.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:30.435+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:30.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:30.436+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:31.455 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:31.197+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:31.456 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:31.197+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:31.456 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:31.197+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:31.456 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:31.198+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:31.456 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:31.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:31.456 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:31.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:31.457 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:31.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:31.457 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:31.171+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:31.706 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:16:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:16:31.584+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 15199 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:16:31.707 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:31.451+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:31.707 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:31.451+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:31.707 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:31.451+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:31.707 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:31.452+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:32.452 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:32.177+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:32.452 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:32.177+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:32.452 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:32.177+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:32.452 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:32.178+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:32.453 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:32.220+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:32.453 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:32.221+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:32.453 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:32.221+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:32.453 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:32.221+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:32.453 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:32.446+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:32.454 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:32.446+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:32.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:32.446+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:32.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:32.448+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:33.498 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:33.423+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:33.498 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:33.423+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:33.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:33.423+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:33.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:33.424+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:33.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:33.181+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:33.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:33.181+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:33.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:33.181+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:33.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:33.182+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:33.500 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:33.221+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:33.500 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:33.221+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:33.500 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:33.221+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:33.500 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:33.222+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:34.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:34.236+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:34.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:34.236+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:34.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:34.236+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:34.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:34.237+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:34.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:34.139+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:34.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:34.139+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:34.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:34.139+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:34.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:34.140+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:34.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:34.435+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:34.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:34.435+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:34.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:34.435+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:34.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:34.436+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:35.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:35.215+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:35.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:35.215+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:35.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:35.215+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:35.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:35.216+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:35.142+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:35.142+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:35.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:35.142+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:35.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:35.142+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:35.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:35.404+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:35.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:35.404+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:35.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:35.404+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:35.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:35.405+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:36.589 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:36.212+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:36.589 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:36.212+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:36.589 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:36.212+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:36.590 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:36.213+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:36.590 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:36.389+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:36.590 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:36.389+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:36.590 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:36.389+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:36.590 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:36.390+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:36.591 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:36.189+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:36.591 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:36.189+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:36.591 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:36.189+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:36.591 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:36.190+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:36.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:16:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:16:36.585+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 15279 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:16:37.437 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:37.252+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:37.437 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:37.252+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:37.437 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:37.252+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:37.437 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:37.254+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:37.438 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:37.154+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:37.438 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:37.154+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:37.438 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:37.154+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:37.438 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:37.155+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:37.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:37.433+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:37.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:37.433+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:37.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:37.433+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:37.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:37.434+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:38.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:38.146+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:38.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:38.146+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:38.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:38.146+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:38.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:38.147+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:38.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:38.385+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:38.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:38.385+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:38.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:38.385+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:38.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:38.386+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:38.410 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:38.228+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:38.410 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:38.228+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:38.410 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:38.228+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:38.410 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:38.229+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:39.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:39.366+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:39.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:39.366+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:39.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:39.366+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:39.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:39.367+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:39.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:39.224+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:39.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:39.224+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:39.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:39.224+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:39.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:39.225+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:39.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:39.192+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:39.660 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:39.192+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:39.660 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:39.192+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:39.660 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:39.193+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:40.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:40.228+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:40.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:40.228+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:40.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:40.228+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:40.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:40.229+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:40.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:40.245+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:40.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:40.245+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:40.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:40.245+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:40.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:40.246+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:40.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:40.347+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:40.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:40.347+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:40.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:40.347+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:40.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:40.348+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:41.591 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:41.288+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:41.591 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:41.288+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:41.591 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:41.288+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:41.591 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:41.289+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:41.591 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:41.310+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:41.592 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:41.310+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:41.592 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:41.310+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:41.592 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:41.311+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:41.592 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:41.251+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:41.592 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:41.251+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:41.592 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:41.251+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:41.593 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:41.252+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:41.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:16:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:16:41.587+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 15362 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:16:42.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:42.337+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:42.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:42.337+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:42.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:42.337+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:42.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:42.338+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:42.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:42.296+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:42.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:42.296+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:42.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:42.296+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:42.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:42.297+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:42.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:42.239+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:42.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:42.239+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:42.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:42.239+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:42.660 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:42.240+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:43.588 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:43.260+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:43.588 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:43.260+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:43.588 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:43.260+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:43.588 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:43.260+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:43.589 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:43.281+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:43.589 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:43.281+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:43.589 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:43.281+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:43.589 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:43.282+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:43.590 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:43.364+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:43.590 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:43.364+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:43.590 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:43.364+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:43.590 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:43.365+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:44.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:44.327+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:44.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:44.327+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:44.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:44.327+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:44.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:44.328+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:44.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:44.365+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:44.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:44.365+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:44.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:44.365+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:44.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:44.366+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:44.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:44.223+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:44.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:44.223+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:44.660 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:44.223+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:44.660 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:44.224+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:45.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:45.303+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:45.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:45.303+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:45.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:45.303+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:45.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:45.304+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:45.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:45.318+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:45.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:45.318+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:45.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:45.318+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:45.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:45.318+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:45.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:45.185+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:45.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:45.185+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:45.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:45.185+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:45.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:45.186+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:46.593 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:46.354+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:46.593 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:46.354+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:46.593 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:46.354+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:46.593 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:46.354+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:46.593 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:46.271+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:46.594 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:46.271+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:46.594 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:46.271+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:46.594 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:46.272+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:46.594 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:46.235+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:46.594 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:46.235+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:46.594 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:46.235+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:46.594 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:46.236+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:46.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:16:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:16:46.588+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 15442 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:16:47.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:47.293+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:47.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:47.293+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:47.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:47.293+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:47.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:47.293+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:47.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:47.263+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:47.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:47.263+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:47.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:47.263+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:47.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:47.263+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:47.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:47.347+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:47.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:47.347+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:47.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:47.347+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:47.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:47.348+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:48.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:48.327+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:48.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:48.327+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:48.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:48.327+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:48.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:48.327+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:48.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:48.215+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:48.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:48.215+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:48.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:48.215+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:48.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:48.216+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:48.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:48.338+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:48.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:48.338+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:48.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:48.338+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:48.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:48.339+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:49.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:49.332+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:49.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:49.332+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:49.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:49.332+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:49.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:49.333+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:49.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:49.233+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:49.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:49.233+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:49.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:49.233+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:49.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:49.234+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:49.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:49.384+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:49.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:49.384+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:49.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:49.384+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:49.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:49.385+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:50.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:50.210+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:50.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:50.210+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:50.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:50.210+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:50.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:50.211+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:50.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:50.358+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:50.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:50.358+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:50.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:50.358+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:50.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:50.359+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:50.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:50.354+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:50.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:50.354+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:50.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:50.354+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:50.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:50.355+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:51.594 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:51.255+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:51.594 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:51.255+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:51.594 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:51.255+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:51.594 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:51.256+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:51.595 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:51.375+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:51.595 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:51.375+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:51.595 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:51.375+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:51.595 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:51.376+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:51.595 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:51.364+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:51.595 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:51.364+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:51.596 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:51.364+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:51.596 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:51.365+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:51.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:16:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:16:51.590+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 15525 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:16:52.372 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:52.282+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:52.372 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:52.282+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:52.372 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:52.282+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:52.373 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:52.282+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:52.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:52.412+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:52.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:52.412+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:52.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:52.412+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:52.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:52.413+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:52.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:52.368+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:52.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:52.368+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:52.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:52.368+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:52.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:52.369+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:53.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:53.235+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:53.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:53.235+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:53.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:53.235+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:53.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:53.236+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:53.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:53.404+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:53.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:53.404+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:53.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:53.404+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:53.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:53.404+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:53.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:53.419+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:53.501 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:53.419+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:53.501 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:53.419+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:53.501 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:53.419+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:54.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:54.244+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:54.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:54.244+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:54.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:54.244+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:54.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:54.244+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:54.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:54.443+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:54.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:54.443+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:54.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:54.443+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:54.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:54.444+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:54.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:54.361+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:54.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:54.361+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:54.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:54.361+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:54.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:54.362+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:55.383 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:55.242+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:55.384 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:55.242+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:55.384 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:55.242+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:55.384 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:55.243+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:55.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:55.379+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:55.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:55.379+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:55.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:55.379+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:55.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:55.380+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:55.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:55.441+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:55.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:55.441+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:55.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:55.441+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:55.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:55.442+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:56.596 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:56.226+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:56.596 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:56.226+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:56.596 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:56.226+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:56.596 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:56.227+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:56.597 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:56.407+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:56.597 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:56.407+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:56.597 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:56.407+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:56.597 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:56.407+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:56.597 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:56.346+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:56.597 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:56.346+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:56.598 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:56.346+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:56.598 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:56.347+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:56.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:16:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:16:56.591+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 15605 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:16:57.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:57.213+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:57.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:57.213+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:57.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:57.213+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:57.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:57.214+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:57.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:57.388+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:57.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:57.388+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:57.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:57.388+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:57.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:57.389+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:57.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:57.371+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:57.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:57.371+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:57.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:57.371+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:57.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:57.372+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:58.559 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:58.237+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:58.559 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:58.237+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:58.559 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:58.237+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:58.559 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:58.238+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:58.560 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:58.358+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:58.560 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:58.358+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:58.560 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:58.358+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:58.560 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:58.359+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:58.560 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:58.411+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:58.561 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:58.411+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:58.561 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:58.411+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:58.561 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:58.412+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:59.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:59.263+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:59.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:59.263+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:59.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:59.263+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:16:59.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:16:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:16:59.264+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:16:59.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:59.319+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:59.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:59.319+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:59.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:59.319+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:16:59.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:16:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:16:59.320+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:16:59.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:59.441+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:59.660 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:59.441+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:59.660 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:59.441+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:16:59.660 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:16:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:16:59.442+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:00.485 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:00.290+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:00.485 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:00.290+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:00.485 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:00.290+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:00.486 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:00.291+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:00.486 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:00.293+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:00.486 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:00.293+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:00.486 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:00.293+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:00.486 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:00.294+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:00.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:00.481+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:00.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:00.481+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:00.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:00.481+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:00.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:00.482+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:01.597 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:01.264+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:01.597 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:01.264+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:01.597 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:01.264+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:01.597 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:01.266+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:01.598 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:01.340+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:01.598 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:01.340+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:01.598 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:01.340+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:01.598 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:01.341+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:01.598 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:01.449+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:01.598 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:01.449+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:01.599 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:01.449+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:01.599 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:01.450+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:01.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:17:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:17:01.593+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 15688 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:17:02.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:02.361+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:02.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:02.361+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:02.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:02.361+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:02.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:02.362+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:02.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:02.415+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:02.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:02.415+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:02.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:02.415+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:02.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:02.416+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:02.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:02.244+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:02.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:02.244+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:02.660 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:02.244+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:02.660 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:02.245+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:03.498 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:03.380+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:03.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:03.380+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:03.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:03.380+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:03.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:03.381+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:03.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:03.379+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:03.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:03.379+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:03.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:03.379+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:03.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:03.380+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:03.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:03.234+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:03.501 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:03.234+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:03.501 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:03.234+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:03.501 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:03.235+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:04.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:04.244+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:04.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:04.244+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:04.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:04.244+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:04.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:04.245+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:04.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:04.367+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:04.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:04.367+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:04.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:04.367+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:04.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:04.368+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:04.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:04.380+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:04.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:04.380+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:04.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:04.380+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:04.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:04.381+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:05.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:05.319+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:05.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:05.319+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:05.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:05.319+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:05.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:05.319+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:05.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:05.262+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:05.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:05.262+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:05.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:05.262+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:05.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:05.263+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:05.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:05.339+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:05.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:05.339+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:05.660 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:05.339+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:05.660 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:05.340+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:06.599 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:06.357+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:06.599 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:06.357+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:06.599 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:06.357+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:06.599 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:06.358+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:06.600 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:06.246+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:06.600 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:06.246+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:06.600 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:06.246+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:06.600 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:06.247+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:06.600 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:06.295+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:06.600 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:06.295+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:06.601 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:06.295+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:06.601 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:06.295+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:06.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:17:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:17:06.594+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 15768 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:17:07.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:07.286+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:07.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:07.286+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:07.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:07.286+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:07.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:07.286+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:07.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:07.271+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:07.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:07.271+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:07.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:07.271+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:07.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:07.271+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:07.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:07.387+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:07.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:07.387+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:07.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:07.387+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:07.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:07.388+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:08.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:08.331+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:08.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:08.331+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:08.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:08.331+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:08.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:08.331+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:08.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:08.233+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:08.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:08.233+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:08.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:08.233+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:08.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:08.234+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:08.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:08.394+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:08.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:08.394+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:08.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:08.394+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:08.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:08.394+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:09.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:09.366+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:09.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:09.366+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:09.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:09.366+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:09.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:09.368+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:09.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:09.241+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:09.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:09.241+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:09.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:09.241+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:09.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:09.241+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:09.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:09.440+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:09.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:09.440+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:09.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:09.440+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:09.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:09.441+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:10.465 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:10.209+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:10.466 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:10.209+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:10.466 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:10.209+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:10.466 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:10.209+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:10.466 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:10.328+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:10.466 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:10.328+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:10.466 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:10.328+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:10.467 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:10.329+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:10.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:10.461+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:10.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:10.461+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:10.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:10.461+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:10.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:10.463+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:11.458 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:11.179+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:11.458 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:11.179+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:11.458 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:11.179+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:11.458 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:11.180+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:11.459 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:11.301+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:11.459 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:11.301+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:11.459 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:11.301+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:11.459 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:11.302+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:11.726 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:17:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:17:11.596+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 15851 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:17:11.726 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:11.454+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:11.726 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:11.454+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:11.727 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:11.454+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:11.727 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:11.455+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:12.480 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:12.298+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:12.480 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:12.298+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:12.480 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:12.298+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:12.480 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:12.299+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:12.480 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:12.188+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:12.481 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:12.188+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:12.481 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:12.188+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:12.481 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:12.189+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:12.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:12.476+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:12.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:12.476+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:12.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:12.476+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:12.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:12.476+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:13.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:13.263+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:13.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:13.263+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:13.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:13.263+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:13.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:13.264+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:13.500 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:13.179+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:13.500 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:13.179+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:13.500 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:13.179+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:13.500 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:13.180+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:13.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:13.512+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:13.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:13.512+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:13.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:13.512+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:13.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:13.513+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:14.498 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:14.157+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:14.498 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:14.157+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:14.498 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:14.157+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:14.498 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:14.158+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:14.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:14.231+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:14.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:14.231+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:14.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:14.231+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:14.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:14.232+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:14.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:14.494+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:14.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:14.494+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:14.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:14.494+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:14.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:14.495+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:15.524 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:15.159+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:15.524 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:15.159+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:15.525 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:15.159+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:15.525 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:15.160+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:15.525 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:15.213+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:15.525 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:15.213+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:15.525 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:15.213+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:15.525 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:15.214+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:15.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:15.520+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:15.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:15.520+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:15.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:15.520+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:15.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:15.521+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:16.480 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:16.196+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:16.480 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:16.196+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:16.480 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:16.196+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:16.480 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:16.196+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:16.480 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:16.221+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:16.481 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:16.222+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:16.481 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:16.222+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:16.481 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:16.222+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:16.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:17:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:17:16.598+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 15931 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:17:16.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:16.476+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:16.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:16.476+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:16.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:16.476+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:16.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:16.477+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:17.451 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:17.174+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:17.451 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:17.174+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:17.451 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:17.174+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:17.452 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:17.175+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:17.452 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:17.232+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:17.452 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:17.232+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:17.452 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:17.232+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:17.452 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:17.234+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:17.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:17.446+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:17.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:17.446+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:17.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:17.446+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:17.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:17.447+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:18.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:18.405+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:18.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:18.405+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:18.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:18.405+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:18.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:18.406+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:18.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:18.179+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:18.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:18.179+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:18.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:18.179+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:18.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:18.180+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:18.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:18.204+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:18.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:18.204+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:18.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:18.204+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:18.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:18.205+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:19.443 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:19.201+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:19.444 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:19.201+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:19.444 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:19.201+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:19.444 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:19.202+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:19.444 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:19.175+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:19.444 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:19.175+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:19.445 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:19.175+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:19.445 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:19.176+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:19.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:19.439+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:19.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:19.439+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:19.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:19.440+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:19.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:19.440+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:20.459 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:20.223+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:20.459 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:20.223+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:20.459 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:20.223+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:20.459 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:20.224+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:20.459 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:20.165+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:20.460 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:20.165+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:20.460 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:20.165+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:20.460 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:20.166+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:20.718 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:20.455+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:20.719 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:20.455+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:20.719 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:20.455+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:20.719 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:20.455+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:21.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:21.117+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:21.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:21.117+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:21.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:21.117+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:21.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:21.118+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:21.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:21.235+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:21.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:21.235+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:21.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:21.235+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:21.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:21.236+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:21.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:17:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:17:21.599+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 16014 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:17:21.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:21.479+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:21.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:21.479+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:21.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:21.479+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:21.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:21.480+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:22.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:22.274+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:22.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:22.274+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:22.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:22.274+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:22.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:22.275+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:22.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:22.068+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:22.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:22.068+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:22.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:22.068+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:22.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:22.069+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:22.766 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:22.478+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:22.766 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:22.478+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:22.766 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:22.478+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:22.766 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:22.479+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:23.318 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:23.049+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:23.318 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:23.049+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:23.319 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:23.049+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:23.319 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:23.049+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:23.589 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:23.314+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:23.589 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:23.314+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:23.589 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:23.314+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:23.589 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:23.315+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:23.590 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:23.494+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:23.590 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:23.494+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:23.590 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:23.494+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:23.590 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:23.495+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:24.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:24.275+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:24.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:24.275+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:24.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:24.275+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:24.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:24.275+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:24.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:24.060+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:24.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:24.060+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:24.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:24.060+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:24.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:24.061+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:24.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:24.484+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:24.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:24.484+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:24.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:24.484+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:24.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:24.484+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:25.302 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:25.042+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:25.302 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:25.042+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:25.302 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:25.042+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:25.302 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:25.043+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:25.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:25.298+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:25.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:25.298+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:25.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:25.298+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:25.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:25.299+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:25.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:25.504+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:25.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:25.504+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:25.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:25.504+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:25.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:25.505+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:26.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:25.995+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:26.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:25.995+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:26.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:25.995+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:26.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:25.996+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:26.605 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:26.525+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:26.605 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:26.525+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:26.605 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:26.525+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:26.605 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:26.525+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:26.606 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:26.297+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:26.606 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:26.297+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:26.606 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:26.297+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:26.606 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:26.297+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:26.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:17:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:17:26.601+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 16094 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:17:27.351 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:26.994+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:27.351 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:26.994+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:27.352 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:26.994+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:27.352 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:26.995+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:27.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:27.506+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:27.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:27.506+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:27.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:27.506+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:27.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:27.507+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:27.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:27.347+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:27.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:27.347+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:27.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:27.347+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:27.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:27.348+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:28.374 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:27.974+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:28.375 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:27.974+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:28.375 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:27.974+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:28.375 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:27.975+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:28.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:28.369+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:28.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:28.369+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:28.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:28.369+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:28.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:28.370+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:28.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:28.458+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:28.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:28.458+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:28.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:28.458+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:28.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:28.459+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:29.376 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:28.942+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:29.376 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:28.942+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:29.376 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:28.942+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:29.376 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:28.943+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:29.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:29.371+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:29.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:29.371+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:29.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:29.371+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:29.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:29.372+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:29.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:29.437+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:29.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:29.437+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:29.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:29.437+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:29.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:29.438+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:30.401 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:29.980+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:30.401 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:29.980+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:30.401 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:29.980+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:30.401 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:29.981+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:30.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:30.396+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:30.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:30.396+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:30.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:30.396+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:30.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:30.397+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:30.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:30.411+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:30.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:30.411+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:30.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:30.411+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:30.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:30.412+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:31.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:30.933+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:31.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:30.933+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:31.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:30.933+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:31.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:30.933+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:31.702 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:17:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:17:31.601+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 16177 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:17:31.703 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:31.429+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:31.703 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:31.429+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:31.703 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:31.429+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:31.703 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:31.430+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:31.704 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:31.450+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:31.704 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:31.450+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:31.704 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:31.450+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:31.704 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:31.450+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:31.982 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:31.933+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:31.982 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:31.933+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:31.982 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:31.933+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:31.982 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:31.934+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:32.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:32.445+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:32.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:32.445+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:32.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:32.445+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:32.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:32.446+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:32.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:32.476+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:32.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:32.476+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:32.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:32.476+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:32.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:32.477+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:33.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:32.933+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:33.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:32.933+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:33.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:32.933+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:33.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:32.934+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:33.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:33.487+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:33.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:33.487+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:33.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:33.487+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:33.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:33.488+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:33.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:33.473+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:33.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:33.473+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:33.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:33.473+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:33.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:33.474+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:34.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:33.933+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:34.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:33.933+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:34.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:33.933+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:34.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:33.934+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:34.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:34.472+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:34.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:34.472+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:34.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:34.472+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:34.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:34.472+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:34.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:34.458+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:34.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:34.458+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:34.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:34.458+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:34.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:34.459+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:35.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:34.952+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:35.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:34.952+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:35.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:34.952+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:35.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:34.952+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:35.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:35.503+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:35.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:35.503+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:35.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:35.503+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:35.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:35.504+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:35.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:35.500+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:35.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:35.500+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:35.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:35.500+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:35.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:35.501+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:36.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:36.003+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:36.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:36.003+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:36.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:36.003+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:36.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:36.004+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:36.733 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:17:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:17:36.604+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 16257 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:17:36.734 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:36.509+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:36.734 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:36.509+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:36.734 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:36.509+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:36.734 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:36.510+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:36.734 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:36.455+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:36.734 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:36.455+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:36.735 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:36.455+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:36.735 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:36.455+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:37.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:37.048+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:37.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:37.048+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:37.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:37.048+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:37.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:37.049+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:37.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:37.560+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:37.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:37.560+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:37.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:37.560+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:37.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:37.560+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:37.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:37.486+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:37.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:37.486+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:37.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:37.486+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:37.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:37.487+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:38.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:38.022+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:38.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:38.022+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:38.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:38.022+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:38.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:38.023+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:38.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:38.477+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:38.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:38.477+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:38.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:38.477+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:38.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:38.477+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:38.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:38.575+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:38.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:38.575+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:38.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:38.575+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:38.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:38.575+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:39.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:39.032+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:39.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:39.032+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:39.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:39.032+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:39.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:39.032+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:39.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:39.566+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:39.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:39.566+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:39.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:39.566+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:39.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:39.567+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:39.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:39.476+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:39.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:39.476+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:39.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:39.476+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:39.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:39.477+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:40.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:40.049+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:40.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:40.049+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:40.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:40.049+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:40.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:40.050+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:40.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:40.467+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:40.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:40.467+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:40.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:40.467+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:40.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:40.468+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:40.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:40.579+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:40.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:40.579+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:40.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:40.579+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:40.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:40.580+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:41.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:41.043+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:41.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:41.043+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:41.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:41.043+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:41.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:41.044+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:41.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:17:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:17:41.605+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 16340 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:17:41.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:41.568+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:41.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:41.568+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:41.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:41.568+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:41.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:41.569+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:41.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:41.509+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:41.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:41.509+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:41.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:41.509+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:41.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:41.510+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:42.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:42.016+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:42.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:42.016+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:42.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:42.016+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:42.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:42.018+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:42.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:42.602+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:42.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:42.602+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:42.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:42.602+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:42.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:42.603+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:42.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:42.536+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:42.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:42.536+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:42.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:42.536+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:42.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:42.537+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:43.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:43.008+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:43.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:43.008+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:43.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:43.008+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:43.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:43.008+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:43.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:43.584+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:43.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:43.584+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:43.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:43.584+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:43.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:43.585+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:43.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:43.557+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:43.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:43.557+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:43.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:43.557+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:43.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:43.558+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:44.152 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-02-09T22:17:44.153 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2024-02-09T22:17:44.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:17:44.153 DEBUG:teuthology.orchestra.run.smithi006:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2024-02-09T22:17:44.191 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_getpshared: Cannot mkdir: Input/output error 2024-02-09T22:17:44.192 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_getpshared/3-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.192 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_getpshared/coverage.txt: Cannot open: Input/output error 2024-02-09T22:17:44.192 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_getpshared/1-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.192 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_getpshared/1-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.192 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_getpshared/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.192 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_getpshared/1-3.c: Cannot open: Input/output error 2024-02-09T22:17:44.192 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_getpshared/2-1.sh: Cannot open: Input/output error 2024-02-09T22:17:44.192 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigignore: Cannot mkdir: Input/output error 2024-02-09T22:17:44.193 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigignore/5-3.sh: Cannot open: Input/output error 2024-02-09T22:17:44.193 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigignore/6-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.193 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigignore/coverage.txt: Cannot open: Input/output error 2024-02-09T22:17:44.193 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigignore/5-1.sh: Cannot open: Input/output error 2024-02-09T22:17:44.193 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigignore/5-core-buildonly.c: Cannot open: Input/output error 2024-02-09T22:17:44.193 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigignore/6-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.193 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigignore/1-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.193 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigignore/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.194 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigignore/4-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.194 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigignore/5-4.sh: Cannot open: Input/output error 2024-02-09T22:17:44.194 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigignore/5-2.sh: Cannot open: Input/output error 2024-02-09T22:17:44.194 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_trywrlock: Cannot mkdir: Input/output error 2024-02-09T22:17:44.194 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_trywrlock/coverage.txt: Cannot open: Input/output error 2024-02-09T22:17:44.194 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_trywrlock/1-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.194 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_trywrlock/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.194 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_trywrlock/2-1.sh: Cannot open: Input/output error 2024-02-09T22:17:44.195 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_trywrlock/speculative: Cannot mkdir: Input/output error 2024-02-09T22:17:44.195 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_trywrlock/speculative/3-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.195 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_barrierattr_getpshared: Cannot mkdir: Input/output error 2024-02-09T22:17:44.195 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_barrierattr_getpshared/coverage.txt: Cannot open: Input/output error 2024-02-09T22:17:44.195 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_barrierattr_getpshared/2-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.195 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_barrierattr_getpshared/1-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.195 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_barrierattr_getpshared/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.195 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_barrierattr_getpshared/3-1.sh: Cannot open: Input/output error 2024-02-09T22:17:44.196 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mlock: Cannot mkdir: Input/output error 2024-02-09T22:17:44.196 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mlock/10-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.196 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mlock/coverage.txt: Cannot open: Input/output error 2024-02-09T22:17:44.196 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mlock/5-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.196 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mlock/8-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.196 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mlock/7-1.sh: Cannot open: Input/output error 2024-02-09T22:17:44.196 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mlock/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.196 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mlock/12-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.197 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mlock/4-1.sh: Cannot open: Input/output error 2024-02-09T22:17:44.197 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mlock/2-1.sh: Cannot open: Input/output error 2024-02-09T22:17:44.197 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mlock/speculative: Cannot mkdir: Input/output error 2024-02-09T22:17:44.197 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mlock/speculative/12-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.197 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_setattr: Cannot mkdir: Input/output error 2024-02-09T22:17:44.197 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_setattr/coverage.txt: Cannot open: Input/output error 2024-02-09T22:17:44.197 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_setattr/1-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.197 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_setattr/2-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.198 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_setattr/5-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.198 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_setattr/1-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.198 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_setattr/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.198 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_setattr/3-1.sh: Cannot open: Input/output error 2024-02-09T22:17:44.198 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_setattr/4-1.sh: Cannot open: Input/output error 2024-02-09T22:17:44.198 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock_nanosleep: Cannot mkdir: Input/output error 2024-02-09T22:17:44.198 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock_nanosleep/3-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.198 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock_nanosleep/9-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.198 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock_nanosleep/10-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.199 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock_nanosleep/15-1.sh: Cannot open: Input/output error 2024-02-09T22:17:44.199 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock_nanosleep/1-2.sh: Cannot open: Input/output error 2024-02-09T22:17:44.199 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock_nanosleep/coverage.txt: Cannot open: Input/output error 2024-02-09T22:17:44.199 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock_nanosleep/5-1.sh: Cannot open: Input/output error 2024-02-09T22:17:44.199 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock_nanosleep/6-1.sh: Cannot open: Input/output error 2024-02-09T22:17:44.199 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock_nanosleep/8-1.sh: Cannot open: Input/output error 2024-02-09T22:17:44.199 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock_nanosleep/2-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.199 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock_nanosleep/1-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.200 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock_nanosleep/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.200 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock_nanosleep/1-5.c: Cannot open: Input/output error 2024-02-09T22:17:44.200 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock_nanosleep/2-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.200 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock_nanosleep/13-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.200 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock_nanosleep/4-1.sh: Cannot open: Input/output error 2024-02-09T22:17:44.201 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock_nanosleep/1-3.c: Cannot open: Input/output error 2024-02-09T22:17:44.201 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock_nanosleep/1-4.c: Cannot open: Input/output error 2024-02-09T22:17:44.201 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock_nanosleep/11-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.201 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock_nanosleep/2-3.c: Cannot open: Input/output error 2024-02-09T22:17:44.201 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_error: Cannot mkdir: Input/output error 2024-02-09T22:17:44.201 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_error/3-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.201 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_error/coverage.txt: Cannot open: Input/output error 2024-02-09T22:17:44.201 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_error/2-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.201 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_error/1-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.202 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_error/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.202 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction: Cannot mkdir: Input/output error 2024-02-09T22:17:44.202 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-16.c: Cannot open: Input/output error 2024-02-09T22:17:44.202 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/8-3.c: Cannot open: Input/output error 2024-02-09T22:17:44.202 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-14.c: Cannot open: Input/output error 2024-02-09T22:17:44.202 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/22-9.c: Cannot open: Input/output error 2024-02-09T22:17:44.202 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/13-25.c: Cannot open: Input/output error 2024-02-09T22:17:44.202 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/6-21.c: Cannot open: Input/output error 2024-02-09T22:17:44.203 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-39.c: Cannot open: Input/output error 2024-02-09T22:17:44.203 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/6-24.c: Cannot open: Input/output error 2024-02-09T22:17:44.203 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/16-7.c: Cannot open: Input/output error 2024-02-09T22:17:44.203 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-79.c: Cannot open: Input/output error 2024-02-09T22:17:44.203 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/16-11.c: Cannot open: Input/output error 2024-02-09T22:17:44.203 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-44.c: Cannot open: Input/output error 2024-02-09T22:17:44.203 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/13-16.c: Cannot open: Input/output error 2024-02-09T22:17:44.203 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/1-6.c: Cannot open: Input/output error 2024-02-09T22:17:44.203 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-50.c: Cannot open: Input/output error 2024-02-09T22:17:44.204 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/templates: Cannot mkdir: Input/output error 2024-02-09T22:17:44.204 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/templates/template_28-1.in: Cannot open: Input/output error 2024-02-09T22:17:44.204 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/templates/template_22-1.in: Cannot open: Input/output error 2024-02-09T22:17:44.204 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/templates/template_2-1.in: Cannot open: Input/output error 2024-02-09T22:17:44.204 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/templates/template_23-1.in: Cannot open: Input/output error 2024-02-09T22:17:44.204 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/templates/template_16-1.in: Cannot open: Input/output error 2024-02-09T22:17:44.204 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/templates/template_4-3.in: Cannot open: Input/output error 2024-02-09T22:17:44.204 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/templates/template_13-1.in: Cannot open: Input/output error 2024-02-09T22:17:44.205 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/templates/template_3-1.in: Cannot open: Input/output error 2024-02-09T22:17:44.205 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/templates/template_25-1.in: Cannot open: Input/output error 2024-02-09T22:17:44.205 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/templates/template_19-1.in: Cannot open: Input/output error 2024-02-09T22:17:44.205 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/templates/template_1-1.in: Cannot open: Input/output error 2024-02-09T22:17:44.205 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/templates/template_4-4.in: Cannot open: Input/output error 2024-02-09T22:17:44.205 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/templates/template_8-1.in: Cannot open: Input/output error 2024-02-09T22:17:44.205 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/templates/template_17-1.in: Cannot open: Input/output error 2024-02-09T22:17:44.205 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/templates/template_6-1.in: Cannot open: Input/output error 2024-02-09T22:17:44.206 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/templates/template_4-1.in: Cannot open: Input/output error 2024-02-09T22:17:44.206 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/templates/template_12-1.in: Cannot open: Input/output error 2024-02-09T22:17:44.206 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/templates/template_18-1.in: Cannot open: Input/output error 2024-02-09T22:17:44.206 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/templates/template_4-2.in: Cannot open: Input/output error 2024-02-09T22:17:44.206 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/templates/template_12-2.in: Cannot open: Input/output error 2024-02-09T22:17:44.206 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-28.c: Cannot open: Input/output error 2024-02-09T22:17:44.206 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-6.c: Cannot open: Input/output error 2024-02-09T22:17:44.206 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-22.c: Cannot open: Input/output error 2024-02-09T22:17:44.207 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/8-14.c: Cannot open: Input/output error 2024-02-09T22:17:44.207 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-89.c: Cannot open: Input/output error 2024-02-09T22:17:44.207 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/19-11.c: Cannot open: Input/output error 2024-02-09T22:17:44.207 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-48.c: Cannot open: Input/output error 2024-02-09T22:17:44.207 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/23-18.c: Cannot open: Input/output error 2024-02-09T22:17:44.207 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-46.c: Cannot open: Input/output error 2024-02-09T22:17:44.207 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/19-20.c: Cannot open: Input/output error 2024-02-09T22:17:44.207 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/17-20.c: Cannot open: Input/output error 2024-02-09T22:17:44.207 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/16-5.c: Cannot open: Input/output error 2024-02-09T22:17:44.207 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/28-19.c: Cannot open: Input/output error 2024-02-09T22:17:44.208 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/22-4.c: Cannot open: Input/output error 2024-02-09T22:17:44.208 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/6-15.c: Cannot open: Input/output error 2024-02-09T22:17:44.208 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/19-7.c: Cannot open: Input/output error 2024-02-09T22:17:44.208 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/3-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.208 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/18-3.c: Cannot open: Input/output error 2024-02-09T22:17:44.208 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-45.c: Cannot open: Input/output error 2024-02-09T22:17:44.208 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/22-15.c: Cannot open: Input/output error 2024-02-09T22:17:44.208 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/25-10.c: Cannot open: Input/output error 2024-02-09T22:17:44.208 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-44.c: Cannot open: Input/output error 2024-02-09T22:17:44.209 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/9-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.209 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-47.c: Cannot open: Input/output error 2024-02-09T22:17:44.209 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/1-24.c: Cannot open: Input/output error 2024-02-09T22:17:44.209 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-103.c: Cannot open: Input/output error 2024-02-09T22:17:44.209 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/25-18.c: Cannot open: Input/output error 2024-02-09T22:17:44.209 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/18-8.c: Cannot open: Input/output error 2024-02-09T22:17:44.209 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/8-26.c: Cannot open: Input/output error 2024-02-09T22:17:44.209 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/2-24.c: Cannot open: Input/output error 2024-02-09T22:17:44.209 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/28-5.c: Cannot open: Input/output error 2024-02-09T22:17:44.210 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/25-16.c: Cannot open: Input/output error 2024-02-09T22:17:44.210 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/2-21.c: Cannot open: Input/output error 2024-02-09T22:17:44.210 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/13-5.c: Cannot open: Input/output error 2024-02-09T22:17:44.210 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/13-21.c: Cannot open: Input/output error 2024-02-09T22:17:44.215 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-48.c: Cannot open: Input/output error 2024-02-09T22:17:44.215 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-12.c: Cannot open: Input/output error 2024-02-09T22:17:44.215 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-60.c: Cannot open: Input/output error 2024-02-09T22:17:44.215 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/28-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.215 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/16-15.c: Cannot open: Input/output error 2024-02-09T22:17:44.215 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-81.c: Cannot open: Input/output error 2024-02-09T22:17:44.215 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/25-22.c: Cannot open: Input/output error 2024-02-09T22:17:44.215 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-5.c: Cannot open: Input/output error 2024-02-09T22:17:44.216 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/25-21.c: Cannot open: Input/output error 2024-02-09T22:17:44.216 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/28-7.c: Cannot open: Input/output error 2024-02-09T22:17:44.216 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-24.c: Cannot open: Input/output error 2024-02-09T22:17:44.216 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.216 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-46.c: Cannot open: Input/output error 2024-02-09T22:17:44.216 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/19-16.c: Cannot open: Input/output error 2024-02-09T22:17:44.216 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/1-23.c: Cannot open: Input/output error 2024-02-09T22:17:44.216 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/18-19.c: Cannot open: Input/output error 2024-02-09T22:17:44.217 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/3-17.c: Cannot open: Input/output error 2024-02-09T22:17:44.217 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/28-6.c: Cannot open: Input/output error 2024-02-09T22:17:44.217 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/1-22.c: Cannot open: Input/output error 2024-02-09T22:17:44.217 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-67.c: Cannot open: Input/output error 2024-02-09T22:17:44.217 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/19-26.c: Cannot open: Input/output error 2024-02-09T22:17:44.217 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/23-11.c: Cannot open: Input/output error 2024-02-09T22:17:44.217 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/10-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.217 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-18.c: Cannot open: Input/output error 2024-02-09T22:17:44.217 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/16-25.c: Cannot open: Input/output error 2024-02-09T22:17:44.218 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/8-23.c: Cannot open: Input/output error 2024-02-09T22:17:44.218 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/2-10.c: Cannot open: Input/output error 2024-02-09T22:17:44.218 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-49.c: Cannot open: Input/output error 2024-02-09T22:17:44.218 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-11.c: Cannot open: Input/output error 2024-02-09T22:17:44.218 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/2-12.c: Cannot open: Input/output error 2024-02-09T22:17:44.218 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/25-13.c: Cannot open: Input/output error 2024-02-09T22:17:44.218 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/13-22.c: Cannot open: Input/output error 2024-02-09T22:17:44.218 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/1-8.c: Cannot open: Input/output error 2024-02-09T22:17:44.218 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/25-15.c: Cannot open: Input/output error 2024-02-09T22:17:44.219 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/28-22.c: Cannot open: Input/output error 2024-02-09T22:17:44.219 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-27.c: Cannot open: Input/output error 2024-02-09T22:17:44.219 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/1-19.c: Cannot open: Input/output error 2024-02-09T22:17:44.219 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/22-10.c: Cannot open: Input/output error 2024-02-09T22:17:44.219 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-29.c: Cannot open: Input/output error 2024-02-09T22:17:44.219 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/13-4.c: Cannot open: Input/output error 2024-02-09T22:17:44.219 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/3-20.c: Cannot open: Input/output error 2024-02-09T22:17:44.219 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-37.c: Cannot open: Input/output error 2024-02-09T22:17:44.219 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-57.c: Cannot open: Input/output error 2024-02-09T22:17:44.220 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-36.c: Cannot open: Input/output error 2024-02-09T22:17:44.220 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/18-20.c: Cannot open: Input/output error 2024-02-09T22:17:44.220 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/3-14.c: Cannot open: Input/output error 2024-02-09T22:17:44.220 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/gentests.pl: Cannot open: Input/output error 2024-02-09T22:17:44.220 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/25-24.c: Cannot open: Input/output error 2024-02-09T22:17:44.221 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-7.c: Cannot open: Input/output error 2024-02-09T22:17:44.221 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/17-9.c: Cannot open: Input/output error 2024-02-09T22:17:44.221 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-02-09T22:17:44.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:17:44.222 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-02-09T22:17:44.280 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-02-09T22:17:44.281 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/17-16.c: Cannot open: Input/output error 2024-02-09T22:17:44.281 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/16-24.c: Cannot open: Input/output error 2024-02-09T22:17:44.281 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/13-9.c: Cannot open: Input/output error 2024-02-09T22:17:44.281 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/23-16.c: Cannot open: Input/output error 2024-02-09T22:17:44.281 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/18-14.c: Cannot open: Input/output error 2024-02-09T22:17:44.281 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/3-5.c: Cannot open: Input/output error 2024-02-09T22:17:44.281 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-27.c: Cannot open: Input/output error 2024-02-09T22:17:44.281 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/19-3.c: Cannot open: Input/output error 2024-02-09T22:17:44.282 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/8-24.c: Cannot open: Input/output error 2024-02-09T22:17:44.282 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/8-13.c: Cannot open: Input/output error 2024-02-09T22:17:44.282 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/22-14.c: Cannot open: Input/output error 2024-02-09T22:17:44.282 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/23-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.282 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/23-19.c: Cannot open: Input/output error 2024-02-09T22:17:44.282 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/23-21.c: Cannot open: Input/output error 2024-02-09T22:17:44.282 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/6-25.c: Cannot open: Input/output error 2024-02-09T22:17:44.282 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/13-7.c: Cannot open: Input/output error 2024-02-09T22:17:44.282 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-102.c: Cannot open: Input/output error 2024-02-09T22:17:44.283 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/18-26.c: Cannot open: Input/output error 2024-02-09T22:17:44.283 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/28-26.c: Cannot open: Input/output error 2024-02-09T22:17:44.283 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-52.c: Cannot open: Input/output error 2024-02-09T22:17:44.283 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-8.c: Cannot open: Input/output error 2024-02-09T22:17:44.283 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/16-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.283 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/16-21.c: Cannot open: Input/output error 2024-02-09T22:17:44.283 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/6-16.c: Cannot open: Input/output error 2024-02-09T22:17:44.283 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/23-8.c: Cannot open: Input/output error 2024-02-09T22:17:44.283 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/6-11.c: Cannot open: Input/output error 2024-02-09T22:17:44.284 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/6-6.c: Cannot open: Input/output error 2024-02-09T22:17:44.284 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/19-12.c: Cannot open: Input/output error 2024-02-09T22:17:44.284 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-94.c: Cannot open: Input/output error 2024-02-09T22:17:44.284 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/28-18.c: Cannot open: Input/output error 2024-02-09T22:17:44.284 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/18-17.c: Cannot open: Input/output error 2024-02-09T22:17:44.284 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/1-26.c: Cannot open: Input/output error 2024-02-09T22:17:44.284 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/28-16.c: Cannot open: Input/output error 2024-02-09T22:17:44.285 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/8-9.c: Cannot open: Input/output error 2024-02-09T22:17:44.285 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/19-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.285 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-71.c: Cannot open: Input/output error 2024-02-09T22:17:44.285 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/22-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.285 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-77.c: Cannot open: Input/output error 2024-02-09T22:17:44.285 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/17-3.c: Cannot open: Input/output error 2024-02-09T22:17:44.285 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/28-15.c: Cannot open: Input/output error 2024-02-09T22:17:44.285 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/13-17.c: Cannot open: Input/output error 2024-02-09T22:17:44.285 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-87.c: Cannot open: Input/output error 2024-02-09T22:17:44.286 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/23-15.c: Cannot open: Input/output error 2024-02-09T22:17:44.286 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/6-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.286 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/18-18.c: Cannot open: Input/output error 2024-02-09T22:17:44.286 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/coverage.txt: Cannot open: Input/output error 2024-02-09T22:17:44.286 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/13-20.c: Cannot open: Input/output error 2024-02-09T22:17:44.286 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/17-5.c: Cannot open: Input/output error 2024-02-09T22:17:44.286 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/23-22.c: Cannot open: Input/output error 2024-02-09T22:17:44.286 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/18-11.c: Cannot open: Input/output error 2024-02-09T22:17:44.286 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/18-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.287 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/13-8.c: Cannot open: Input/output error 2024-02-09T22:17:44.287 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/6-4.c: Cannot open: Input/output error 2024-02-09T22:17:44.287 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-19.c: Cannot open: Input/output error 2024-02-09T22:17:44.287 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/8-4.c: Cannot open: Input/output error 2024-02-09T22:17:44.287 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-25.c: Cannot open: Input/output error 2024-02-09T22:17:44.287 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/6-8.c: Cannot open: Input/output error 2024-02-09T22:17:44.287 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/25-14.c: Cannot open: Input/output error 2024-02-09T22:17:44.287 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/6-26.c: Cannot open: Input/output error 2024-02-09T22:17:44.287 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/13-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.288 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/28-9.c: Cannot open: Input/output error 2024-02-09T22:17:44.288 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-10.c: Cannot open: Input/output error 2024-02-09T22:17:44.288 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-74.c: Cannot open: Input/output error 2024-02-09T22:17:44.288 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/6-20.c: Cannot open: Input/output error 2024-02-09T22:17:44.288 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/16-4.c: Cannot open: Input/output error 2024-02-09T22:17:44.288 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/17-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.288 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/17-25.c: Cannot open: Input/output error 2024-02-09T22:17:44.288 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/23-3.c: Cannot open: Input/output error 2024-02-09T22:17:44.289 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/16-13.c: Cannot open: Input/output error 2024-02-09T22:17:44.289 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/19-23.c: Cannot open: Input/output error 2024-02-09T22:17:44.289 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/13-26.c: Cannot open: Input/output error 2024-02-09T22:17:44.289 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/19-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.289 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/6-5.c: Cannot open: Input/output error 2024-02-09T22:17:44.289 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-88.c: Cannot open: Input/output error 2024-02-09T22:17:44.289 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/28-3.c: Cannot open: Input/output error 2024-02-09T22:17:44.289 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-50.c: Cannot open: Input/output error 2024-02-09T22:17:44.289 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/17-22.c: Cannot open: Input/output error 2024-02-09T22:17:44.290 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/25-4.c: Cannot open: Input/output error 2024-02-09T22:17:44.290 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-11.c: Cannot open: Input/output error 2024-02-09T22:17:44.290 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-23.c: Cannot open: Input/output error 2024-02-09T22:17:44.290 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-33.c: Cannot open: Input/output error 2024-02-09T22:17:44.290 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/23-14.c: Cannot open: Input/output error 2024-02-09T22:17:44.290 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/17-4.c: Cannot open: Input/output error 2024-02-09T22:17:44.290 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/28-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.290 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/1-16.c: Cannot open: Input/output error 2024-02-09T22:17:44.291 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/18-22.c: Cannot open: Input/output error 2024-02-09T22:17:44.291 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-98.c: Cannot open: Input/output error 2024-02-09T22:17:44.291 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/1-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.291 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/25-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.291 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/6-9.c: Cannot open: Input/output error 2024-02-09T22:17:44.291 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/testfrmw.c: Cannot open: Input/output error 2024-02-09T22:17:44.291 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/17-7.c: Cannot open: Input/output error 2024-02-09T22:17:44.291 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/22-18.c: Cannot open: Input/output error 2024-02-09T22:17:44.291 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/13-23.c: Cannot open: Input/output error 2024-02-09T22:17:44.292 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/16-20.c: Cannot open: Input/output error 2024-02-09T22:17:44.292 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-31.c: Cannot open: Input/output error 2024-02-09T22:17:44.292 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/8-25.c: Cannot open: Input/output error 2024-02-09T22:17:44.292 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-41.c: Cannot open: Input/output error 2024-02-09T22:17:44.292 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/testfrmw.h: Cannot open: Input/output error 2024-02-09T22:17:44.292 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-54.c: Cannot open: Input/output error 2024-02-09T22:17:44.292 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-30.c: Cannot open: Input/output error 2024-02-09T22:17:44.292 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/2-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.293 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-26.c: Cannot open: Input/output error 2024-02-09T22:17:44.293 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/6-22.c: Cannot open: Input/output error 2024-02-09T22:17:44.293 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/16-10.c: Cannot open: Input/output error 2024-02-09T22:17:44.293 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/3-12.c: Cannot open: Input/output error 2024-02-09T22:17:44.293 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-16.c: Cannot open: Input/output error 2024-02-09T22:17:44.293 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/16-17.c: Cannot open: Input/output error 2024-02-09T22:17:44.293 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-4.c: Cannot open: Input/output error 2024-02-09T22:17:44.293 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-91.c: Cannot open: Input/output error 2024-02-09T22:17:44.293 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-18.c: Cannot open: Input/output error 2024-02-09T22:17:44.293 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-20.c: Cannot open: Input/output error 2024-02-09T22:17:44.294 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/6-14.c: Cannot open: Input/output error 2024-02-09T22:17:44.294 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/1-7.c: Cannot open: Input/output error 2024-02-09T22:17:44.294 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-17.c: Cannot open: Input/output error 2024-02-09T22:17:44.294 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/8-17.c: Cannot open: Input/output error 2024-02-09T22:17:44.294 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/22-24.c: Cannot open: Input/output error 2024-02-09T22:17:44.294 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-99.c: Cannot open: Input/output error 2024-02-09T22:17:44.294 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/6-23.c: Cannot open: Input/output error 2024-02-09T22:17:44.294 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/17-23.c: Cannot open: Input/output error 2024-02-09T22:17:44.294 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/6-10.c: Cannot open: Input/output error 2024-02-09T22:17:44.294 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-49.c: Cannot open: Input/output error 2024-02-09T22:17:44.295 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/6-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.295 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-13.c: Cannot open: Input/output error 2024-02-09T22:17:44.295 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-100.c: Cannot open: Input/output error 2024-02-09T22:17:44.295 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/3-18.c: Cannot open: Input/output error 2024-02-09T22:17:44.295 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/25-9.c: Cannot open: Input/output error 2024-02-09T22:17:44.295 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/1-15.c: Cannot open: Input/output error 2024-02-09T22:17:44.295 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/18-23.c: Cannot open: Input/output error 2024-02-09T22:17:44.295 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/23-20.c: Cannot open: Input/output error 2024-02-09T22:17:44.295 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/23-23.c: Cannot open: Input/output error 2024-02-09T22:17:44.296 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/22-25.c: Cannot open: Input/output error 2024-02-09T22:17:44.296 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-19.c: Cannot open: Input/output error 2024-02-09T22:17:44.296 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/1-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.296 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-68.c: Cannot open: Input/output error 2024-02-09T22:17:44.296 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/16-8.c: Cannot open: Input/output error 2024-02-09T22:17:44.296 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/18-21.c: Cannot open: Input/output error 2024-02-09T22:17:44.296 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/23-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.296 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/8-12.c: Cannot open: Input/output error 2024-02-09T22:17:44.296 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-53.c: Cannot open: Input/output error 2024-02-09T22:17:44.297 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-69.c: Cannot open: Input/output error 2024-02-09T22:17:44.297 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-30.c: Cannot open: Input/output error 2024-02-09T22:17:44.297 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-28.c: Cannot open: Input/output error 2024-02-09T22:17:44.297 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/3-22.c: Cannot open: Input/output error 2024-02-09T22:17:44.297 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/16-16.c: Cannot open: Input/output error 2024-02-09T22:17:44.297 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/22-17.c: Cannot open: Input/output error 2024-02-09T22:17:44.297 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-62.c: Cannot open: Input/output error 2024-02-09T22:17:44.297 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-36.c: Cannot open: Input/output error 2024-02-09T22:17:44.297 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/19-17.c: Cannot open: Input/output error 2024-02-09T22:17:44.298 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/18-24.c: Cannot open: Input/output error 2024-02-09T22:17:44.298 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/19-4.c: Cannot open: Input/output error 2024-02-09T22:17:44.298 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-97.c: Cannot open: Input/output error 2024-02-09T22:17:44.298 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-101.c: Cannot open: Input/output error 2024-02-09T22:17:44.298 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/17-15.c: Cannot open: Input/output error 2024-02-09T22:17:44.298 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/23-25.c: Cannot open: Input/output error 2024-02-09T22:17:44.298 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/17-18.c: Cannot open: Input/output error 2024-02-09T22:17:44.298 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/28-12.c: Cannot open: Input/output error 2024-02-09T22:17:44.298 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/25-8.c: Cannot open: Input/output error 2024-02-09T22:17:44.299 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-61.c: Cannot open: Input/output error 2024-02-09T22:17:44.299 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-40.c: Cannot open: Input/output error 2024-02-09T22:17:44.299 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/3-24.c: Cannot open: Input/output error 2024-02-09T22:17:44.299 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/16-9.c: Cannot open: Input/output error 2024-02-09T22:17:44.299 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-84.c: Cannot open: Input/output error 2024-02-09T22:17:44.299 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/17-17.c: Cannot open: Input/output error 2024-02-09T22:17:44.299 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-80.c: Cannot open: Input/output error 2024-02-09T22:17:44.299 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/22-20.c: Cannot open: Input/output error 2024-02-09T22:17:44.299 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-42.c: Cannot open: Input/output error 2024-02-09T22:17:44.300 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/17-26.c: Cannot open: Input/output error 2024-02-09T22:17:44.300 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/28-8.c: Cannot open: Input/output error 2024-02-09T22:17:44.300 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/17-21.c: Cannot open: Input/output error 2024-02-09T22:17:44.300 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/22-16.c: Cannot open: Input/output error 2024-02-09T22:17:44.300 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-70.c: Cannot open: Input/output error 2024-02-09T22:17:44.300 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-76.c: Cannot open: Input/output error 2024-02-09T22:17:44.300 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/28-10.c: Cannot open: Input/output error 2024-02-09T22:17:44.300 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/3-25.c: Cannot open: Input/output error 2024-02-09T22:17:44.300 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/25-7.c: Cannot open: Input/output error 2024-02-09T22:17:44.301 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/3-11.c: Cannot open: Input/output error 2024-02-09T22:17:44.301 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/README: Cannot open: Input/output error 2024-02-09T22:17:44.301 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-25.c: Cannot open: Input/output error 2024-02-09T22:17:44.301 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/22-3.c: Cannot open: Input/output error 2024-02-09T22:17:44.301 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-92.c: Cannot open: Input/output error 2024-02-09T22:17:44.301 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/28-11.c: Cannot open: Input/output error 2024-02-09T22:17:44.301 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/17-24.c: Cannot open: Input/output error 2024-02-09T22:17:44.301 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/16-6.c: Cannot open: Input/output error 2024-02-09T22:17:44.301 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/2-18.c: Cannot open: Input/output error 2024-02-09T22:17:44.301 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-63.c: Cannot open: Input/output error 2024-02-09T22:17:44.302 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-9.c: Cannot open: Input/output error 2024-02-09T22:17:44.302 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/8-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.302 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/17-11.c: Cannot open: Input/output error 2024-02-09T22:17:44.302 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/22-21.c: Cannot open: Input/output error 2024-02-09T22:17:44.302 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/13-19.c: Cannot open: Input/output error 2024-02-09T22:17:44.302 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-40.c: Cannot open: Input/output error 2024-02-09T22:17:44.302 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/2-6.c: Cannot open: Input/output error 2024-02-09T22:17:44.302 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/18-12.c: Cannot open: Input/output error 2024-02-09T22:17:44.303 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-58.c: Cannot open: Input/output error 2024-02-09T22:17:44.303 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/16-19.c: Cannot open: Input/output error 2024-02-09T22:17:44.303 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/8-5.c: Cannot open: Input/output error 2024-02-09T22:17:44.303 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/25-26.c: Cannot open: Input/output error 2024-02-09T22:17:44.303 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-43.c: Cannot open: Input/output error 2024-02-09T22:17:44.303 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/2-20.c: Cannot open: Input/output error 2024-02-09T22:17:44.303 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-55.c: Cannot open: Input/output error 2024-02-09T22:17:44.303 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/1-25.c: Cannot open: Input/output error 2024-02-09T22:17:44.303 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-31.c: Cannot open: Input/output error 2024-02-09T22:17:44.304 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/18-16.c: Cannot open: Input/output error 2024-02-09T22:17:44.304 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/2-7.c: Cannot open: Input/output error 2024-02-09T22:17:44.304 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/1-18.c: Cannot open: Input/output error 2024-02-09T22:17:44.304 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/16-12.c: Cannot open: Input/output error 2024-02-09T22:17:44.304 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/17-19.c: Cannot open: Input/output error 2024-02-09T22:17:44.304 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/1-9.c: Cannot open: Input/output error 2024-02-09T22:17:44.304 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/25-25.c: Cannot open: Input/output error 2024-02-09T22:17:44.304 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/13-18.c: Cannot open: Input/output error 2024-02-09T22:17:44.305 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/3-7.c: Cannot open: Input/output error 2024-02-09T22:17:44.305 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/22-11.c: Cannot open: Input/output error 2024-02-09T22:17:44.305 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/6-7.c: Cannot open: Input/output error 2024-02-09T22:17:44.305 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-38.c: Cannot open: Input/output error 2024-02-09T22:17:44.305 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.305 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-82.c: Cannot open: Input/output error 2024-02-09T22:17:44.305 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/3-15.c: Cannot open: Input/output error 2024-02-09T22:17:44.305 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/1-10.c: Cannot open: Input/output error 2024-02-09T22:17:44.306 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/17-14.c: Cannot open: Input/output error 2024-02-09T22:17:44.306 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/22-8.c: Cannot open: Input/output error 2024-02-09T22:17:44.306 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-95.c: Cannot open: Input/output error 2024-02-09T22:17:44.306 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/8-15.c: Cannot open: Input/output error 2024-02-09T22:17:44.306 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/23-9.c: Cannot open: Input/output error 2024-02-09T22:17:44.306 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-17.c: Cannot open: Input/output error 2024-02-09T22:17:44.306 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/23-17.c: Cannot open: Input/output error 2024-02-09T22:17:44.306 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/13-15.c: Cannot open: Input/output error 2024-02-09T22:17:44.306 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/8-22.c: Cannot open: Input/output error 2024-02-09T22:17:44.307 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/2-22.c: Cannot open: Input/output error 2024-02-09T22:17:44.307 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-5.c: Cannot open: Input/output error 2024-02-09T22:17:44.307 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.307 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/8-7.c: Cannot open: Input/output error 2024-02-09T22:17:44.307 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/16-23.c: Cannot open: Input/output error 2024-02-09T22:17:44.307 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-51.c: Cannot open: Input/output error 2024-02-09T22:17:44.307 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/8-8.c: Cannot open: Input/output error 2024-02-09T22:17:44.307 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/3-16.c: Cannot open: Input/output error 2024-02-09T22:17:44.307 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/18-13.c: Cannot open: Input/output error 2024-02-09T22:17:44.308 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/19-15.c: Cannot open: Input/output error 2024-02-09T22:17:44.308 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/3-13.c: Cannot open: Input/output error 2024-02-09T22:17:44.308 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/21-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.308 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/2-26.c: Cannot open: Input/output error 2024-02-09T22:17:44.308 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-86.c: Cannot open: Input/output error 2024-02-09T22:17:44.308 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-7.c: Cannot open: Input/output error 2024-02-09T22:17:44.308 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/3-6.c: Cannot open: Input/output error 2024-02-09T22:17:44.308 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/18-25.c: Cannot open: Input/output error 2024-02-09T22:17:44.309 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-78.c: Cannot open: Input/output error 2024-02-09T22:17:44.309 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/23-6.c: Cannot open: Input/output error 2024-02-09T22:17:44.309 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-3.c: Cannot open: Input/output error 2024-02-09T22:17:44.309 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/22-5.c: Cannot open: Input/output error 2024-02-09T22:17:44.309 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-93.c: Cannot open: Input/output error 2024-02-09T22:17:44.309 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/2-14.c: Cannot open: Input/output error 2024-02-09T22:17:44.309 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/23-7.c: Cannot open: Input/output error 2024-02-09T22:17:44.309 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/1-5.c: Cannot open: Input/output error 2024-02-09T22:17:44.309 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/25-17.c: Cannot open: Input/output error 2024-02-09T22:17:44.310 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-13.c: Cannot open: Input/output error 2024-02-09T22:17:44.310 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-37.c: Cannot open: Input/output error 2024-02-09T22:17:44.310 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-104.c: Cannot open: Input/output error 2024-02-09T22:17:44.310 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/8-20.c: Cannot open: Input/output error 2024-02-09T22:17:44.310 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/8-18.c: Cannot open: Input/output error 2024-02-09T22:17:44.310 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/13-3.c: Cannot open: Input/output error 2024-02-09T22:17:44.310 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/29-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.310 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/17-12.c: Cannot open: Input/output error 2024-02-09T22:17:44.311 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/28-25.c: Cannot open: Input/output error 2024-02-09T22:17:44.311 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/1-20.c: Cannot open: Input/output error 2024-02-09T22:17:44.311 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/25-19.c: Cannot open: Input/output error 2024-02-09T22:17:44.311 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-38.c: Cannot open: Input/output error 2024-02-09T22:17:44.311 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/28-4.c: Cannot open: Input/output error 2024-02-09T22:17:44.311 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/3-8.c: Cannot open: Input/output error 2024-02-09T22:17:44.311 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/22-12.c: Cannot open: Input/output error 2024-02-09T22:17:44.311 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-52.c: Cannot open: Input/output error 2024-02-09T22:17:44.311 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-3.c: Cannot open: Input/output error 2024-02-09T22:17:44.312 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-39.c: Cannot open: Input/output error 2024-02-09T22:17:44.312 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-33.c: Cannot open: Input/output error 2024-02-09T22:17:44.312 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/19-22.c: Cannot open: Input/output error 2024-02-09T22:17:44.312 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/2-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.312 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-34.c: Cannot open: Input/output error 2024-02-09T22:17:44.312 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/6-12.c: Cannot open: Input/output error 2024-02-09T22:17:44.312 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/3-10.c: Cannot open: Input/output error 2024-02-09T22:17:44.312 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/16-18.c: Cannot open: Input/output error 2024-02-09T22:17:44.313 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/19-24.c: Cannot open: Input/output error 2024-02-09T22:17:44.313 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/25-3.c: Cannot open: Input/output error 2024-02-09T22:17:44.313 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/2-23.c: Cannot open: Input/output error 2024-02-09T22:17:44.313 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/22-7.c: Cannot open: Input/output error 2024-02-09T22:17:44.313 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/17-6.c: Cannot open: Input/output error 2024-02-09T22:17:44.313 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/28-14.c: Cannot open: Input/output error 2024-02-09T22:17:44.313 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/8-19.c: Cannot open: Input/output error 2024-02-09T22:17:44.313 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/2-4.c: Cannot open: Input/output error 2024-02-09T22:17:44.313 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-15.c: Cannot open: Input/output error 2024-02-09T22:17:44.314 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-4.c: Cannot open: Input/output error 2024-02-09T22:17:44.314 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-35.c: Cannot open: Input/output error 2024-02-09T22:17:44.314 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/13-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.314 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/28-23.c: Cannot open: Input/output error 2024-02-09T22:17:44.314 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/1-17.c: Cannot open: Input/output error 2024-02-09T22:17:44.314 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/23-4.c: Cannot open: Input/output error 2024-02-09T22:17:44.314 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/2-11.c: Cannot open: Input/output error 2024-02-09T22:17:44.314 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-21.c: Cannot open: Input/output error 2024-02-09T22:17:44.314 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/2-16.c: Cannot open: Input/output error 2024-02-09T22:17:44.315 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-42.c: Cannot open: Input/output error 2024-02-09T22:17:44.315 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/2-9.c: Cannot open: Input/output error 2024-02-09T22:17:44.315 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-8.c: Cannot open: Input/output error 2024-02-09T22:17:44.315 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/22-6.c: Cannot open: Input/output error 2024-02-09T22:17:44.315 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/28-21.c: Cannot open: Input/output error 2024-02-09T22:17:44.315 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-22.c: Cannot open: Input/output error 2024-02-09T22:17:44.315 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-85.c: Cannot open: Input/output error 2024-02-09T22:17:44.315 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/23-10.c: Cannot open: Input/output error 2024-02-09T22:17:44.315 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/28-24.c: Cannot open: Input/output error 2024-02-09T22:17:44.316 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-14.c: Cannot open: Input/output error 2024-02-09T22:17:44.316 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/22-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.316 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-26.c: Cannot open: Input/output error 2024-02-09T22:17:44.316 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/3-19.c: Cannot open: Input/output error 2024-02-09T22:17:44.316 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/13-12.c: Cannot open: Input/output error 2024-02-09T22:17:44.316 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-6.c: Cannot open: Input/output error 2024-02-09T22:17:44.316 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/19-18.c: Cannot open: Input/output error 2024-02-09T22:17:44.316 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-15.c: Cannot open: Input/output error 2024-02-09T22:17:44.316 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/25-23.c: Cannot open: Input/output error 2024-02-09T22:17:44.317 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/22-19.c: Cannot open: Input/output error 2024-02-09T22:17:44.317 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-72.c: Cannot open: Input/output error 2024-02-09T22:17:44.317 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/1-12.c: Cannot open: Input/output error 2024-02-09T22:17:44.317 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/13-11.c: Cannot open: Input/output error 2024-02-09T22:17:44.317 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/19-21.c: Cannot open: Input/output error 2024-02-09T22:17:44.317 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.317 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/8-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.317 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/13-6.c: Cannot open: Input/output error 2024-02-09T22:17:44.317 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/19-8.c: Cannot open: Input/output error 2024-02-09T22:17:44.318 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/1-3.c: Cannot open: Input/output error 2024-02-09T22:17:44.318 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/19-14.c: Cannot open: Input/output error 2024-02-09T22:17:44.318 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/23-26.c: Cannot open: Input/output error 2024-02-09T22:17:44.318 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-32.c: Cannot open: Input/output error 2024-02-09T22:17:44.318 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-90.c: Cannot open: Input/output error 2024-02-09T22:17:44.318 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/19-13.c: Cannot open: Input/output error 2024-02-09T22:17:44.318 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/18-4.c: Cannot open: Input/output error 2024-02-09T22:17:44.318 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/18-7.c: Cannot open: Input/output error 2024-02-09T22:17:44.319 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/19-25.c: Cannot open: Input/output error 2024-02-09T22:17:44.319 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/22-22.c: Cannot open: Input/output error 2024-02-09T22:17:44.319 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/18-6.c: Cannot open: Input/output error 2024-02-09T22:17:44.319 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/23-5.c: Cannot open: Input/output error 2024-02-09T22:17:44.319 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-45.c: Cannot open: Input/output error 2024-02-09T22:17:44.319 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/8-11.c: Cannot open: Input/output error 2024-02-09T22:17:44.319 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/6-13.c: Cannot open: Input/output error 2024-02-09T22:17:44.319 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/1-4.c: Cannot open: Input/output error 2024-02-09T22:17:44.319 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/6-17.c: Cannot open: Input/output error 2024-02-09T22:17:44.320 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/2-17.c: Cannot open: Input/output error 2024-02-09T22:17:44.320 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/1-21.c: Cannot open: Input/output error 2024-02-09T22:17:44.320 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/2-25.c: Cannot open: Input/output error 2024-02-09T22:17:44.320 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/25-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.320 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/3-4.c: Cannot open: Input/output error 2024-02-09T22:17:44.320 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/19-5.c: Cannot open: Input/output error 2024-02-09T22:17:44.320 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/2-13.c: Cannot open: Input/output error 2024-02-09T22:17:44.320 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-24.c: Cannot open: Input/output error 2024-02-09T22:17:44.321 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/19-19.c: Cannot open: Input/output error 2024-02-09T22:17:44.321 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/17-13.c: Cannot open: Input/output error 2024-02-09T22:17:44.321 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/28-20.c: Cannot open: Input/output error 2024-02-09T22:17:44.321 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/23-12.c: Cannot open: Input/output error 2024-02-09T22:17:44.321 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-47.c: Cannot open: Input/output error 2024-02-09T22:17:44.321 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-9.c: Cannot open: Input/output error 2024-02-09T22:17:44.321 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/2-5.c: Cannot open: Input/output error 2024-02-09T22:17:44.321 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/30-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.321 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/18-15.c: Cannot open: Input/output error 2024-02-09T22:17:44.322 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/25-12.c: Cannot open: Input/output error 2024-02-09T22:17:44.322 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/6-18.c: Cannot open: Input/output error 2024-02-09T22:17:44.322 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/1-13.c: Cannot open: Input/output error 2024-02-09T22:17:44.322 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/16-26.c: Cannot open: Input/output error 2024-02-09T22:17:44.322 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/19-6.c: Cannot open: Input/output error 2024-02-09T22:17:44.322 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-23.c: Cannot open: Input/output error 2024-02-09T22:17:44.322 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/25-5.c: Cannot open: Input/output error 2024-02-09T22:17:44.322 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/18-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.323 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-73.c: Cannot open: Input/output error 2024-02-09T22:17:44.323 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-51.c: Cannot open: Input/output error 2024-02-09T22:17:44.323 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-29.c: Cannot open: Input/output error 2024-02-09T22:17:44.323 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/8-21.c: Cannot open: Input/output error 2024-02-09T22:17:44.323 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/16-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.323 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/6-19.c: Cannot open: Input/output error 2024-02-09T22:17:44.323 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-75.c: Cannot open: Input/output error 2024-02-09T22:17:44.323 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/22-13.c: Cannot open: Input/output error 2024-02-09T22:17:44.323 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-35.c: Cannot open: Input/output error 2024-02-09T22:17:44.324 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/11-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.324 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-12.c: Cannot open: Input/output error 2024-02-09T22:17:44.324 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/17-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.324 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/18-10.c: Cannot open: Input/output error 2024-02-09T22:17:44.324 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/25-6.c: Cannot open: Input/output error 2024-02-09T22:17:44.324 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/13-13.c: Cannot open: Input/output error 2024-02-09T22:17:44.324 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/3-23.c: Cannot open: Input/output error 2024-02-09T22:17:44.324 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/25-11.c: Cannot open: Input/output error 2024-02-09T22:17:44.324 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/19-10.c: Cannot open: Input/output error 2024-02-09T22:17:44.325 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-10.c: Cannot open: Input/output error 2024-02-09T22:17:44.325 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-83.c: Cannot open: Input/output error 2024-02-09T22:17:44.325 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/3-9.c: Cannot open: Input/output error 2024-02-09T22:17:44.325 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/17-8.c: Cannot open: Input/output error 2024-02-09T22:17:44.325 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/3-3.c: Cannot open: Input/output error 2024-02-09T22:17:44.325 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/13-10.c: Cannot open: Input/output error 2024-02-09T22:17:44.325 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/23-24.c: Cannot open: Input/output error 2024-02-09T22:17:44.325 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/18-9.c: Cannot open: Input/output error 2024-02-09T22:17:44.325 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/2-8.c: Cannot open: Input/output error 2024-02-09T22:17:44.326 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/23-13.c: Cannot open: Input/output error 2024-02-09T22:17:44.326 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/18-5.c: Cannot open: Input/output error 2024-02-09T22:17:44.326 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/6-3.c: Cannot open: Input/output error 2024-02-09T22:17:44.326 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-56.c: Cannot open: Input/output error 2024-02-09T22:17:44.326 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/13-14.c: Cannot open: Input/output error 2024-02-09T22:17:44.326 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/25-20.c: Cannot open: Input/output error 2024-02-09T22:17:44.326 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/1-11.c: Cannot open: Input/output error 2024-02-09T22:17:44.326 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/16-14.c: Cannot open: Input/output error 2024-02-09T22:17:44.327 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/16-3.c: Cannot open: Input/output error 2024-02-09T22:17:44.327 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/28-13.c: Cannot open: Input/output error 2024-02-09T22:17:44.327 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-64.c: Cannot open: Input/output error 2024-02-09T22:17:44.327 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/19-9.c: Cannot open: Input/output error 2024-02-09T22:17:44.327 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/8-10.c: Cannot open: Input/output error 2024-02-09T22:17:44.327 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/16-22.c: Cannot open: Input/output error 2024-02-09T22:17:44.327 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/3-21.c: Cannot open: Input/output error 2024-02-09T22:17:44.327 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/3-26.c: Cannot open: Input/output error 2024-02-09T22:17:44.327 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-41.c: Cannot open: Input/output error 2024-02-09T22:17:44.328 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-32.c: Cannot open: Input/output error 2024-02-09T22:17:44.328 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/2-15.c: Cannot open: Input/output error 2024-02-09T22:17:44.328 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/13-24.c: Cannot open: Input/output error 2024-02-09T22:17:44.328 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-20.c: Cannot open: Input/output error 2024-02-09T22:17:44.328 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/22-26.c: Cannot open: Input/output error 2024-02-09T22:17:44.328 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/2-19.c: Cannot open: Input/output error 2024-02-09T22:17:44.328 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-96.c: Cannot open: Input/output error 2024-02-09T22:17:44.328 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.329 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/22-23.c: Cannot open: Input/output error 2024-02-09T22:17:44.329 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/8-6.c: Cannot open: Input/output error 2024-02-09T22:17:44.329 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-59.c: Cannot open: Input/output error 2024-02-09T22:17:44.329 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-66.c: Cannot open: Input/output error 2024-02-09T22:17:44.329 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/1-14.c: Cannot open: Input/output error 2024-02-09T22:17:44.329 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/12-43.c: Cannot open: Input/output error 2024-02-09T22:17:44.329 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/2-3.c: Cannot open: Input/output error 2024-02-09T22:17:44.329 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/8-16.c: Cannot open: Input/output error 2024-02-09T22:17:44.330 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/3-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.330 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/28-17.c: Cannot open: Input/output error 2024-02-09T22:17:44.330 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-34.c: Cannot open: Input/output error 2024-02-09T22:17:44.330 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-65.c: Cannot open: Input/output error 2024-02-09T22:17:44.330 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/17-10.c: Cannot open: Input/output error 2024-02-09T22:17:44.330 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigaction/4-21.c: Cannot open: Input/output error 2024-02-09T22:17:44.330 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_init: Cannot mkdir: Input/output error 2024-02-09T22:17:44.330 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_init/3-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.330 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_init/5-3.c: Cannot open: Input/output error 2024-02-09T22:17:44.331 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_init/coverage.txt: Cannot open: Input/output error 2024-02-09T22:17:44.331 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_init/1-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.331 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_init/testfrmw.c: Cannot open: Input/output error 2024-02-09T22:17:44.331 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_init/testfrmw.h: Cannot open: Input/output error 2024-02-09T22:17:44.331 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_init/2-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.331 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_init/5-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.331 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_init/1-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.331 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_init/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.332 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_init/4-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.332 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_init/speculative: Cannot mkdir: Input/output error 2024-02-09T22:17:44.332 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_init/speculative/5-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.332 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_init/3-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.332 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedsend: Cannot mkdir: Input/output error 2024-02-09T22:17:44.332 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedsend/3-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.332 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedsend/9-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.332 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedsend/5-3.c: Cannot open: Input/output error 2024-02-09T22:17:44.333 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedsend/4-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.333 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedsend/20-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.333 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedsend/10-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.333 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedsend/19-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.333 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedsend/coverage.txt: Cannot open: Input/output error 2024-02-09T22:17:44.333 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedsend/18-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.333 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedsend/14-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.333 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedsend/2-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.333 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedsend/5-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.334 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedsend/6-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.334 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedsend/1-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.334 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedsend/5-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.334 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedsend/15-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.334 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedsend/8-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.334 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedsend/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.334 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedsend/12-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.334 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedsend/7-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.334 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedsend/4-3.c: Cannot open: Input/output error 2024-02-09T22:17:44.335 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedsend/13-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.335 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedsend/11-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.335 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedsend/4-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.335 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedsend/16-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.335 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedsend/11-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.335 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedsend/17-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.335 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedsend/speculative: Cannot mkdir: Input/output error 2024-02-09T22:17:44.335 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedsend/speculative/18-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.335 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_timedsend/3-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.336 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/killpg: Cannot mkdir: Input/output error 2024-02-09T22:17:44.336 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/killpg/coverage.txt: Cannot open: Input/output error 2024-02-09T22:17:44.336 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/killpg/1-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.336 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/killpg/2-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.336 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/killpg/5-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.336 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/killpg/6-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.336 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/killpg/1-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.336 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/killpg/8-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.336 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/killpg/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.337 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/killpg/4-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.337 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_cleanup_push: Cannot mkdir: Input/output error 2024-02-09T22:17:44.337 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_cleanup_push/coverage.txt: Cannot open: Input/output error 2024-02-09T22:17:44.337 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_cleanup_push/1-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.337 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_cleanup_push/1-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.337 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_cleanup_push/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.337 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_cleanup_push/1-3.c: Cannot open: Input/output error 2024-02-09T22:17:44.337 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_cleanup_push/2-1.sh: Cannot open: Input/output error 2024-02-09T22:17:44.338 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/nanosleep: Cannot mkdir: Input/output error 2024-02-09T22:17:44.338 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/nanosleep/3-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.338 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/nanosleep/coverage.txt: Cannot open: Input/output error 2024-02-09T22:17:44.338 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/nanosleep/1-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.338 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/nanosleep/2-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.338 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/nanosleep/5-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.338 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/nanosleep/6-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.338 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/nanosleep/1-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.338 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/nanosleep/5-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.339 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/nanosleep/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.339 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/nanosleep/7-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.339 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/nanosleep/4-1.sh: Cannot open: Input/output error 2024-02-09T22:17:44.339 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/nanosleep/1-3.c: Cannot open: Input/output error 2024-02-09T22:17:44.339 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/nanosleep/10000-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.339 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/nanosleep/7-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.339 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/nanosleep/3-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.339 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open: Cannot mkdir: Input/output error 2024-02-09T22:17:44.339 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/27-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.340 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/3-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.340 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/9-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.340 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/9-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.340 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/20-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.340 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/10-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.340 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/23-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.340 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/19-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.340 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/24-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.341 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/coverage.txt: Cannot open: Input/output error 2024-02-09T22:17:44.341 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/18-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.341 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/28-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.341 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/25-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.341 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/14-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.341 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/2-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.341 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/1-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.341 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/27-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.341 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/15-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.342 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/8-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.342 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.342 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/12-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.342 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/7-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.342 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/21-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.342 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/29-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.342 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/13-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.342 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/22-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.342 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/4-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.343 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/8-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.343 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/25-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.343 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/30-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.343 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/16-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.343 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/11-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.343 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/7-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.343 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/17-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.343 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/7-3.c: Cannot open: Input/output error 2024-02-09T22:17:44.344 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/speculative: Cannot mkdir: Input/output error 2024-02-09T22:17:44.344 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/speculative/6-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.344 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/speculative/2-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.344 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/speculative/26-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.344 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_open/speculative/2-3.c: Cannot open: Input/output error 2024-02-09T22:17:44.344 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setdetachstate: Cannot mkdir: Input/output error 2024-02-09T22:17:44.344 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setdetachstate/coverage.txt: Cannot open: Input/output error 2024-02-09T22:17:44.344 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setdetachstate/1-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.345 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setdetachstate/2-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.345 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setdetachstate/1-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.345 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setdetachstate/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.345 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setdetachstate/3-1.sh: Cannot open: Input/output error 2024-02-09T22:17:44.345 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setdetachstate/4-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.345 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_create: Cannot mkdir: Input/output error 2024-02-09T22:17:44.345 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_create/3-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.345 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_create/9-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.345 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_create/10-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.346 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_create/13-1.sh: Cannot open: Input/output error 2024-02-09T22:17:44.346 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_create/coverage.txt: Cannot open: Input/output error 2024-02-09T22:17:44.346 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_create/14-1.sh: Cannot open: Input/output error 2024-02-09T22:17:44.346 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_create/6-1.sh: Cannot open: Input/output error 2024-02-09T22:17:44.346 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_create/1-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.346 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_create/8-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.346 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_create/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.346 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_create/7-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.347 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_create/4-1.sh: Cannot open: Input/output error 2024-02-09T22:17:44.347 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_create/16-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.347 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_create/11-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.347 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_create/speculative: Cannot mkdir: Input/output error 2024-02-09T22:17:44.347 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_create/speculative/2-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.347 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_create/speculative/5-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.347 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_create/speculative/15-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.347 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_lock: Cannot mkdir: Input/output error 2024-02-09T22:17:44.347 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_lock/3-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.348 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_lock/coverage.txt: Cannot open: Input/output error 2024-02-09T22:17:44.348 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_lock/testfrmw.c: Cannot open: Input/output error 2024-02-09T22:17:44.348 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_lock/testfrmw.h: Cannot open: Input/output error 2024-02-09T22:17:44.348 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_lock/2-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.348 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_lock/5-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.348 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_lock/1-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.348 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_lock/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.348 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_lock/4-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.349 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_getprioceiling: Cannot mkdir: Input/output error 2024-02-09T22:17:44.349 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_getprioceiling/coverage.txt: Cannot open: Input/output error 2024-02-09T22:17:44.349 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_getprioceiling/1-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.349 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_getprioceiling/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.349 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_close: Cannot mkdir: Input/output error 2024-02-09T22:17:44.349 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_close/3-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.349 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_close/coverage.txt: Cannot open: Input/output error 2024-02-09T22:17:44.349 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_close/2-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.350 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_close/5-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.350 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_close/1-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.350 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_close/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.350 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_close/4-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.350 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_close/3-3.c: Cannot open: Input/output error 2024-02-09T22:17:44.350 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mq_close/3-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.350 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_getscheduler: Cannot mkdir: Input/output error 2024-02-09T22:17:44.350 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_getscheduler/3-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.350 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_getscheduler/coverage.txt: Cannot open: Input/output error 2024-02-09T22:17:44.351 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_getscheduler/6-1.sh: Cannot open: Input/output error 2024-02-09T22:17:44.351 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_getscheduler/2-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.351 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_getscheduler/5-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.351 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_getscheduler/1-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.351 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_getscheduler/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.351 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_getscheduler/7-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.351 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_getscheduler/4-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.351 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setstackaddr: Cannot mkdir: Input/output error 2024-02-09T22:17:44.351 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setstackaddr/coverage.txt: Cannot open: Input/output error 2024-02-09T22:17:44.352 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_setstackaddr/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.352 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_getparam: Cannot mkdir: Input/output error 2024-02-09T22:17:44.352 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_getparam/3-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.352 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_getparam/coverage.txt: Cannot open: Input/output error 2024-02-09T22:17:44.352 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_getparam/5-1.sh: Cannot open: Input/output error 2024-02-09T22:17:44.352 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_getparam/2-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.352 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_getparam/6-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.352 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_getparam/1-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.353 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_getparam/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.353 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_getparam/4-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.353 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_getparam/speculative: Cannot mkdir: Input/output error 2024-02-09T22:17:44.353 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_getparam/speculative/7-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.353 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mlockall: Cannot mkdir: Input/output error 2024-02-09T22:17:44.353 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mlockall/9-1.sh: Cannot open: Input/output error 2024-02-09T22:17:44.353 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mlockall/coverage.txt: Cannot open: Input/output error 2024-02-09T22:17:44.353 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mlockall/13-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.353 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mlockall/15-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.354 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mlockall/8-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.354 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mlockall/7-1.sh: Cannot open: Input/output error 2024-02-09T22:17:44.354 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mlockall/3-7.c: Cannot open: Input/output error 2024-02-09T22:17:44.354 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mlockall/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.354 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mlockall/3-6.c: Cannot open: Input/output error 2024-02-09T22:17:44.354 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mlockall/13-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.354 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mlockall/2-1.sh: Cannot open: Input/output error 2024-02-09T22:17:44.354 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mlockall/speculative: Cannot mkdir: Input/output error 2024-02-09T22:17:44.354 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/mlockall/speculative/15-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.355 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_gettype: Cannot mkdir: Input/output error 2024-02-09T22:17:44.355 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_gettype/coverage.txt: Cannot open: Input/output error 2024-02-09T22:17:44.355 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_gettype/1-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.355 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_gettype/1-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.355 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_gettype/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.355 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_gettype/1-5.c: Cannot open: Input/output error 2024-02-09T22:17:44.355 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_gettype/1-3.c: Cannot open: Input/output error 2024-02-09T22:17:44.355 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_gettype/1-4.c: Cannot open: Input/output error 2024-02-09T22:17:44.355 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_gettype/2.sh: Cannot open: Input/output error 2024-02-09T22:17:44.356 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_gettype/speculative: Cannot mkdir: Input/output error 2024-02-09T22:17:44.356 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_gettype/speculative/3-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.356 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/fork: Cannot mkdir: Input/output error 2024-02-09T22:17:44.356 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/fork/3-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.356 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/fork/9-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.356 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/fork/19-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.356 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/fork/18-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.356 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/fork/17-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.356 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/fork/testfrmw.c: Cannot open: Input/output error 2024-02-09T22:17:44.357 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/fork/testfrmw.h: Cannot open: Input/output error 2024-02-09T22:17:44.357 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/fork/14-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.357 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/fork/2-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.357 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/fork/6-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.357 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/fork/1-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.357 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/fork/8-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.357 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/fork/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.357 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/fork/12-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.357 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/fork/7-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.358 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/fork/21-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.358 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/fork/13-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.358 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/fork/22-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.358 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/fork/4-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.358 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/fork/16-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.358 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/fork/11-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.358 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/fork/17-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.358 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/fork/messcat_src.txt: Cannot open: Input/output error 2024-02-09T22:17:44.359 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_timedwait: Cannot mkdir: Input/output error 2024-02-09T22:17:44.359 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_timedwait/3-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.359 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_timedwait/9-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.359 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_timedwait/10-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.359 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_timedwait/6-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.359 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_timedwait/coverage.txt: Cannot open: Input/output error 2024-02-09T22:17:44.359 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_timedwait/2-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.359 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_timedwait/6-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.359 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_timedwait/1-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.360 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_timedwait/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.360 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_timedwait/7-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.360 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_timedwait/2-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.360 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_timedwait/4-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.360 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_timedwait/11-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.360 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/difftime: Cannot mkdir: Input/output error 2024-02-09T22:17:44.360 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/difftime/coverage.txt: Cannot open: Input/output error 2024-02-09T22:17:44.363 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/difftime/1-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.364 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/difftime/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.364 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/difftime/2-1.sh: Cannot open: Input/output error 2024-02-09T22:17:44.364 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_cond_wait: Cannot mkdir: Input/output error 2024-02-09T22:17:44.364 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_cond_wait/3-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.364 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_cond_wait/coverage.txt: Cannot open: Input/output error 2024-02-09T22:17:44.364 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_cond_wait/testfrmw.c: Cannot open: Input/output error 2024-02-09T22:17:44.364 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_cond_wait/testfrmw.h: Cannot open: Input/output error 2024-02-09T22:17:44.364 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_cond_wait/2-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.365 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_cond_wait/1-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.365 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_cond_wait/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.365 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_cond_wait/2-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.365 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_cond_wait/4-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.365 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_cond_wait/2-3.c: Cannot open: Input/output error 2024-02-09T22:17:44.365 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_kill: Cannot mkdir: Input/output error 2024-02-09T22:17:44.365 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_kill/3-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.365 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_kill/coverage.txt: Cannot open: Input/output error 2024-02-09T22:17:44.366 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_kill/1-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.366 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_kill/testfrmw.c: Cannot open: Input/output error 2024-02-09T22:17:44.366 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_kill/testfrmw.h: Cannot open: Input/output error 2024-02-09T22:17:44.366 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_kill/2-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.366 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_kill/6-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.366 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_kill/1-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.366 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_kill/8-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.366 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_kill/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.366 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_kill/7-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.367 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_write: Cannot mkdir: Input/output error 2024-02-09T22:17:44.367 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_write/3-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.367 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_write/9-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.367 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_write/9-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.367 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_write/10-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.367 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_write/coverage.txt: Cannot open: Input/output error 2024-02-09T22:17:44.367 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_write/1-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.367 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_write/2-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.368 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_write/5-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.368 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_write/6-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.368 INFO:tasks.workunit.client.0.smithi006.stderr: 2024-02-09T22:17:44.368 INFO:tasks.workunit.client.0.smithi006.stderr:gzip: tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_write/1-1.c: Cannot openstdin: Stale file handle 2024-02-09T22:17:44.368 INFO:tasks.workunit.client.0.smithi006.stderr:: Input/output error 2024-02-09T22:17:44.368 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_write/8-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.368 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_write/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.368 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_write/12-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.368 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_write/7-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.369 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_write/13-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.369 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_write/4-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.369 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_write/8-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.369 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/aio_write/11-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.369 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock_getres: Cannot mkdir: Input/output error 2024-02-09T22:17:44.369 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock_getres/3-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.369 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock_getres/6-2.c: Cannot open: Input/output error 2024-02-09T22:17:44.369 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock_getres/coverage.txt: Cannot open: Input/output error 2024-02-09T22:17:44.369 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock_getres/5-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.370 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock_getres/6-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.370 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock_getres/1-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.370 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock_getres/8-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.370 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock_getres/assertions.xml: Cannot open: Input/output error 2024-02-09T22:17:44.370 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock_getres/7-1.c: Cannot open: Input/output error 2024-02-09T22:17:44.370 INFO:tasks.workunit.client.0.smithi006.stderr:tar: Unexpected EOF in archive 2024-02-09T22:17:44.370 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces: Cannot utime: Input/output error 2024-02-09T22:17:44.370 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance: Cannot utime: Input/output error 2024-02-09T22:17:44.371 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite: Cannot utime: Input/output error 2024-02-09T22:17:44.371 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231/testcases: Cannot utime: Input/output error 2024-02-09T22:17:44.371 INFO:tasks.workunit.client.0.smithi006.stderr:tar: ltp-full-20091231: Cannot utime: Input/output error 2024-02-09T22:17:44.371 INFO:tasks.workunit.client.0.smithi006.stderr:tar: Error is not recoverable: exiting now 2024-02-09T22:17:44.371 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-02-09T22:17:44.372 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-02-09T22:17:44.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:44.021+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:44.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:44.021+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:44.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:44.021+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:44.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:44.022+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:44.535 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-09T22:17:44.535 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-09T22:17:44.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:44.607+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:44.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:44.607+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:44.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:44.607+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:44.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:44.608+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:44.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:44.540+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:44.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:44.540+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:44.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:44.540+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:44.910 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:44.541+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:45.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:45.032+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:45.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:45.032+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:45.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:45.032+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:45.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:45.033+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:45.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:45.625+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:45.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:45.625+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:45.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:45.625+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:45.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:45.626+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:45.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:45.511+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:45.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:45.511+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:45.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:45.511+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:45.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:45.512+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:46.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:46.049+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:46.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:46.050+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:46.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:46.050+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:46.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:46.051+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:46.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:17:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:17:46.607+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 16420 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:17:46.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:46.626+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:46.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:46.626+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:46.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:46.626+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:46.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:46.627+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:46.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:46.522+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:46.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:46.522+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:46.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:46.522+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:46.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:46.522+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:47.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:47.042+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:47.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:47.042+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:47.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:47.042+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:47.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:47.043+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:47.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:47.544+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:47.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:47.544+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:47.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:47.544+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:47.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:47.545+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:47.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:47.642+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:47.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:47.642+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:47.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:47.642+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:47.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:47.643+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:48.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:48.068+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:48.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:48.068+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:48.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:48.068+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:48.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:48.069+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:48.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:48.612+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:48.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:48.612+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:48.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:48.612+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:48.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:48.613+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:48.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:48.565+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:48.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:48.565+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:48.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:48.565+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:48.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:48.565+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:49.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:49.072+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:49.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:49.072+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:49.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:49.072+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:49.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:49.074+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:49.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:49.635+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:49.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:49.635+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:49.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:49.635+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:49.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:49.636+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:49.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:49.578+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:49.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:49.578+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:49.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:49.578+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:49.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:49.579+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:50.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:50.047+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:50.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:50.047+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:50.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:50.047+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:50.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:50.048+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:50.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:50.604+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:50.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:50.604+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:50.911 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:50.604+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:50.911 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:50.605+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:50.911 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:50.536+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:50.912 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:50.536+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:50.912 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:50.536+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:50.912 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:50.537+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:51.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:51.073+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:51.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:51.073+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:51.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:51.073+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:51.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:51.074+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:51.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:17:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:17:51.609+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 16503 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:17:51.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:51.631+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:51.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:51.631+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:51.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:51.631+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:51.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:51.632+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:51.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:51.532+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:51.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:51.532+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:51.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:51.532+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:51.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:51.533+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:52.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:52.053+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:52.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:52.053+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:52.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:52.053+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:52.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:52.054+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:52.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:52.596+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:52.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:52.596+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:52.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:52.596+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:52.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:52.596+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:52.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:52.522+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:52.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:52.522+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:52.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:52.522+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:52.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:52.523+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:53.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:53.028+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:53.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:53.028+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:53.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:53.028+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:53.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:53.028+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:53.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:53.604+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:53.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:53.604+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:53.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:53.604+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:53.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:53.605+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:53.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:53.486+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:53.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:53.486+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:53.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:53.486+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:53.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:53.487+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:54.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:53.983+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:54.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:53.983+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:54.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:53.983+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:54.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:53.984+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:54.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:54.605+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:54.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:54.605+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:54.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:54.605+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:54.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:54.606+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:54.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:54.493+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:54.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:54.493+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:54.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:54.493+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:54.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:54.493+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:55.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:55.008+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:55.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:55.008+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:55.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:55.008+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:55.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:55.008+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:55.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:55.629+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:55.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:55.629+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:55.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:55.629+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:55.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:55.630+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:55.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:55.469+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:55.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:55.469+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:55.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:55.469+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:55.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:55.470+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:56.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:56.024+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:56.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:56.024+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:56.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:56.024+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:56.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:56.025+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:56.738 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:17:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:17:56.610+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 16583 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:17:56.738 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:56.649+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:56.738 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:56.649+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:56.739 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:56.649+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:56.739 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:56.650+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:56.739 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:56.448+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:56.739 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:56.448+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:56.739 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:56.448+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:56.739 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:56.449+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:57.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:57.024+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:57.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:57.024+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:57.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:57.024+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:57.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:57.025+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:57.681 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:57.415+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:57.682 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:57.415+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:57.682 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:57.415+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:57.682 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:57.415+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:58.030 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:57.677+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:58.030 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:57.677+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:58.031 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:57.677+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:58.031 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:57.678+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:58.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:58.026+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:58.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:58.026+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:58.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:58.026+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:58.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:58.027+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:58.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:58.632+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:58.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:58.632+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:58.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:58.632+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:58.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:58.633+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:58.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:58.432+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:58.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:58.432+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:58.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:58.432+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:58.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:58.433+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:59.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:58.995+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:59.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:58.995+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:59.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:58.995+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:17:59.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:17:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:17:58.995+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:17:59.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:59.447+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:59.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:59.447+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:59.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:59.447+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:17:59.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:17:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:17:59.448+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:17:59.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:59.647+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:59.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:59.647+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:59.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:59.647+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:17:59.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:17:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:17:59.647+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:00.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:00.010+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:00.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:00.010+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:00.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:00.010+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:00.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:00.010+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:00.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:00.602+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:00.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:00.602+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:00.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:00.602+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:00.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:00.602+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:00.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:00.472+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:00.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:00.472+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:00.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:00.472+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:00.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:00.473+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:01.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:01.023+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:01.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:01.023+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:01.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:01.023+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:01.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:01.024+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:01.742 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:18:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:18:01.612+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 16667 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:18:01.742 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:01.573+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:01.742 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:01.573+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:01.743 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:01.573+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:01.743 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:01.574+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:01.743 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:01.458+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:01.743 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:01.458+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:01.743 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:01.458+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:01.743 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:01.458+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:02.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:02.074+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:02.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:02.074+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:02.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:02.074+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:02.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:02.075+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:02.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:02.471+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:02.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:02.471+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:02.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:02.471+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:02.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:02.472+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:02.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:02.582+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:02.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:02.582+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:02.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:02.582+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:02.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:02.583+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:03.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:03.107+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:03.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:03.107+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:03.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:03.107+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:03.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:03.108+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:03.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:03.451+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:03.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:03.451+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:03.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:03.451+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:03.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:03.451+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:03.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:03.551+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:03.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:03.551+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:03.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:03.551+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:03.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:03.551+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:04.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:04.143+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:04.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:04.143+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:04.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:04.143+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:04.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:04.144+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:04.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:04.432+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:04.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:04.432+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:04.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:04.432+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:04.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:04.432+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:04.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:04.591+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:04.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:04.591+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:04.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:04.591+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:04.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:04.592+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:05.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:05.108+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:05.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:05.108+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:05.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:05.108+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:05.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:05.109+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:05.774 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:05.560+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:05.774 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:05.560+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:05.774 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:05.560+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:05.774 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:05.561+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:05.775 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:05.449+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:05.775 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:05.449+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:05.775 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:05.449+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:05.775 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:05.450+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:06.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:06.112+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:06.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:06.112+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:06.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:06.112+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:06.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:06.112+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:06.743 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:18:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:18:06.614+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 16747 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:18:06.743 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:06.522+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:06.743 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:06.522+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:06.744 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:06.522+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:06.744 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:06.522+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:06.744 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:06.437+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:06.744 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:06.437+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:06.744 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:06.437+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:06.745 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:06.438+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:07.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:07.086+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:07.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:07.086+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:07.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:07.086+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:07.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:07.086+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:07.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:07.460+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:07.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:07.460+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:07.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:07.460+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:07.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:07.461+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:07.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:07.494+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:07.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:07.494+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:07.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:07.494+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:07.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:07.495+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:08.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:08.089+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:08.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:08.089+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:08.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:08.089+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:08.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:08.090+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:08.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:08.526+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:08.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:08.526+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:08.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:08.526+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:08.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:08.527+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:08.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:08.462+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:08.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:08.462+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:08.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:08.462+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:08.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:08.463+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:09.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:09.098+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:09.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:09.098+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:09.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:09.098+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:09.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:09.098+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:09.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:09.545+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:09.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:09.545+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:09.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:09.545+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:09.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:09.546+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:09.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:09.436+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:09.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:09.436+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:09.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:09.436+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:09.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:09.436+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:10.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:10.077+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:10.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:10.077+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:10.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:10.077+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:10.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:10.078+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:10.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:10.497+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:10.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:10.497+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:10.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:10.497+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:10.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:10.498+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:10.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:10.453+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:10.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:10.453+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:10.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:10.453+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:10.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:10.454+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:11.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:11.127+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:11.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:11.127+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:11.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:11.127+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:11.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:11.128+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:11.705 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:18:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:18:11.615+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 16830 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:18:11.706 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:11.487+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:11.706 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:11.487+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:11.707 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:11.487+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:11.707 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:11.487+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:11.707 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:11.423+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:11.708 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:11.423+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:11.708 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:11.423+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:11.708 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:11.424+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:12.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:12.151+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:12.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:12.151+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:12.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:12.151+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:12.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:12.152+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:12.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:12.521+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:12.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:12.521+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:12.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:12.521+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:12.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:12.522+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:12.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:12.448+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:12.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:12.448+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:12.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:12.448+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:12.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:12.449+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:13.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:13.153+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:13.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:13.153+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:13.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:13.153+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:13.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:13.153+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:13.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:13.546+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:13.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:13.546+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:13.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:13.546+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:13.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:13.546+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:13.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:13.435+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:13.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:13.435+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:13.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:13.435+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:13.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:13.436+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:14.444 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:14.166+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:14.444 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:14.166+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:14.444 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:14.166+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:14.444 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:14.166+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:14.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:14.440+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:14.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:14.440+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:14.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:14.440+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:14.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:14.441+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:14.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:14.581+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:14.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:14.581+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:14.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:14.581+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:14.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:14.582+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:15.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:15.117+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:15.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:15.117+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:15.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:15.117+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:15.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:15.118+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:15.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:15.538+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:15.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:15.538+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:15.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:15.538+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:15.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:15.539+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:15.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:15.481+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:15.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:15.481+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:15.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:15.481+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:15.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:15.482+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:16.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:16.090+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:16.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:16.090+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:16.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:16.090+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:16.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:16.091+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:16.745 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:18:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:18:16.617+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 16911 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:18:16.745 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:16.571+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:16.745 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:16.571+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:16.746 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:16.571+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:16.746 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:16.572+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:16.746 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:16.449+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:16.746 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:16.449+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:16.746 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:16.449+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:16.746 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:16.450+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:17.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:17.125+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:17.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:17.125+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:17.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:17.125+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:17.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:17.126+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:17.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:17.574+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:17.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:17.574+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:17.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:17.574+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:17.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:17.575+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:17.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:17.461+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:17.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:17.461+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:17.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:17.461+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:17.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:17.462+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:18.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:18.114+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:18.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:18.114+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:18.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:18.114+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:18.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:18.115+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:18.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:18.550+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:18.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:18.550+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:18.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:18.550+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:18.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:18.551+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:18.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:18.454+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:18.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:18.454+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:18.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:18.454+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:18.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:18.455+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:19.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:19.076+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:19.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:19.076+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:19.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:19.076+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:19.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:19.077+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:19.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:19.599+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:19.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:19.599+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:19.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:19.599+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:19.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:19.600+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:19.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:19.431+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:19.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:19.431+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:19.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:19.431+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:19.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:19.432+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:20.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:20.040+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:20.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:20.040+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:20.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:20.040+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:20.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:20.041+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:20.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:20.634+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:20.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:20.634+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:20.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:20.634+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:20.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:20.634+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:20.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:20.411+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:20.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:20.411+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:20.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:20.411+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:20.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:20.411+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:21.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:21.003+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:21.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:21.003+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:21.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:21.003+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:21.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:21.004+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:21.705 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:18:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:18:21.619+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 16995 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:18:21.706 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:21.661+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:21.706 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:21.661+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:21.706 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:21.661+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:21.706 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:21.662+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:21.707 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:21.438+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:21.707 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:21.438+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:21.707 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:21.438+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:21.707 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:21.439+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:22.003+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:22.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:22.003+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:22.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:22.003+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:22.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:22.004+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:22.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:22.613+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:22.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:22.613+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:22.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:22.613+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:22.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:22.613+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:22.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:22.463+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:22.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:22.463+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:22.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:22.463+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:22.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:22.464+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:23.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:22.975+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:23.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:22.975+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:23.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:22.975+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:23.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:22.976+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:23.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:23.629+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:23.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:23.629+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:23.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:23.629+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:23.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:23.630+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:23.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:23.476+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:23.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:23.476+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:23.911 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:23.476+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:23.911 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:23.477+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:24.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:23.968+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:24.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:23.968+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:24.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:23.968+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:24.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:23.969+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:24.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:24.587+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:24.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:24.587+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:24.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:24.587+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:24.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:24.588+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:24.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:24.433+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:24.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:24.433+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:24.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:24.433+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:24.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:24.434+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:25.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:25.007+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:25.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:25.007+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:25.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:25.007+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:25.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:25.008+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:25.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:25.596+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:25.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:25.596+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:25.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:25.596+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:25.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:25.597+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:25.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:25.432+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:25.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:25.432+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:25.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:25.432+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:25.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:25.433+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:26.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:26.024+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:26.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:26.024+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:26.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:26.024+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:26.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:26.025+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:26.749 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:18:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:18:26.620+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 17075 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:18:26.750 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:26.554+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:26.750 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:26.554+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:26.750 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:26.554+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:26.750 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:26.555+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:26.750 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:26.417+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:26.751 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:26.417+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:26.751 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:26.417+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:26.751 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:26.417+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:27.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:27.033+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:27.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:27.033+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:27.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:27.033+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:27.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:27.033+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:27.906 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:27.536+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:27.906 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:27.536+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:27.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:27.536+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:27.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:27.537+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:27.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:27.451+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:27.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:27.451+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:27.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:27.451+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:27.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:27.452+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:28.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:27.984+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:28.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:27.984+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:28.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:27.984+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:28.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:27.985+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:28.580 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:28.542+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:28.580 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:28.542+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:28.580 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:28.542+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:28.580 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:28.543+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:28.580 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:28.412+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:28.581 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:28.412+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:28.581 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:28.412+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:28.581 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:28.413+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:29.379 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:29.009+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:29.379 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:29.009+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:29.379 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:29.009+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:29.379 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:29.010+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:29.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:29.537+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:29.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:29.537+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:29.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:29.537+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:29.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:29.537+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:29.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:29.375+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:29.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:29.375+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:29.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:29.375+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:29.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:29.376+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:30.347 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:29.985+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:30.347 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:29.985+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:30.347 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:29.986+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:30.347 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:29.986+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:30.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:30.343+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:30.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:30.343+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:30.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:30.343+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:30.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:30.344+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:30.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:30.575+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:30.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:30.575+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:30.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:30.575+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:30.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:30.575+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:31.309 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:31.000+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:31.310 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:31.000+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:31.310 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:31.000+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:31.310 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:31.001+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:31.310 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:31.305+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:31.310 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:31.305+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:31.310 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:31.305+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:31.626 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:31.306+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:31.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:18:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:18:31.622+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 17158 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:18:31.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:31.625+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:31.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:31.625+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:31.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:31.625+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:31.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:31.626+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:32.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:32.289+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:32.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:32.289+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:32.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:32.289+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:32.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:32.290+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:32.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:32.045+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:32.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:32.045+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:32.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:32.045+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:32.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:32.046+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:32.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:32.650+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:32.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:32.650+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:32.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:32.650+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:32.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:32.651+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:33.336 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:33.018+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:33.337 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:33.018+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:33.337 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:33.018+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:33.337 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:33.019+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:33.595 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:33.332+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:33.595 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:33.332+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:33.595 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:33.332+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:33.595 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:33.333+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:33.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:33.667+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:33.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:33.667+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:33.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:33.667+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:33.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:33.667+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:34.323 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:34.055+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:34.324 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:34.055+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:34.324 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:34.055+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:34.324 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:34.056+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:34.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:34.319+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:34.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:34.319+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:34.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:34.319+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:34.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:34.320+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:35.055 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:34.709+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:35.056 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:34.709+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:35.056 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:34.709+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:35.056 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:34.709+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:35.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:35.283+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:35.283+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:35.283+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:35.284+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:35.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:35.051+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:35.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:35.051+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:35.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:35.051+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:35.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:35.053+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:36.095 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:35.748+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:36.096 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:35.748+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:36.096 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:35.748+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:36.096 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:35.749+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:36.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:36.091+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:36.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:36.091+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:36.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:36.091+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:36.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:36.093+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:36.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:36.256+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:36.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:36.256+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:36.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:36.256+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:36.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:36.257+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:36.756 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:18:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:18:36.624+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 17238 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:18:37.082 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:36.787+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:37.083 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:36.787+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:37.083 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:36.787+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:37.083 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:36.788+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:37.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:37.212+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:37.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:37.212+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:37.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:37.212+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:37.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:37.213+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:37.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:37.078+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:37.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:37.078+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:37.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:37.078+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:37.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:37.079+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:38.121 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:37.826+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:38.121 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:37.826+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:38.121 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:37.826+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:38.121 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:37.827+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:38.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:38.117+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:38.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:38.117+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:38.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:38.117+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:38.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:38.118+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:38.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:38.192+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:38.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:38.192+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:38.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:38.192+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:38.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:38.193+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:38.922 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:38.848+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:38.923 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:38.848+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:38.923 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:38.848+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:38.923 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:38.850+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:39.227 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:39.082+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:39.227 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:39.082+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:39.227 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:39.082+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:39.227 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:39.082+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:39.521 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:39.223+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:39.522 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:39.223+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:39.522 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:39.223+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:39.522 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:39.224+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:40.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:39.859+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:40.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:39.859+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:40.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:39.860+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:40.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:39.860+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:40.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:40.035+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:40.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:40.035+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:40.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:40.035+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:40.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:40.036+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:40.604 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:40.229+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:40.604 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:40.229+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:40.604 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:40.229+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:40.605 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:40.230+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:41.235 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:40.906+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:41.236 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:40.906+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:41.236 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:40.906+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:41.236 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:40.907+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:41.236 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:40.992+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:41.236 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:40.992+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:41.237 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:40.992+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:41.237 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:40.994+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:41.630 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:41.231+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:41.630 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:41.231+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:41.630 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:41.231+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:41.630 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:41.232+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:41.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:18:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:18:41.626+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 17319 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:18:41.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:41.858+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:41.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:41.858+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:41.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:41.858+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:41.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:41.859+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:42.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:42.042+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:42.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:42.042+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:42.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:42.042+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:42.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:42.043+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:42.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:42.213+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:42.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:42.213+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:42.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:42.213+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:42.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:42.214+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:42.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:42.839+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:42.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:42.839+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:42.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:42.839+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:42.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:42.840+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:43.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:43.069+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:43.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:43.069+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:43.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:43.070+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:43.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:43.071+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:43.208+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:43.208+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:43.208+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:43.208+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:44.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:43.817+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:44.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:43.817+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:44.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:43.817+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:44.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:43.818+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:44.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:44.065+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:44.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:44.065+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:44.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:44.065+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:44.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:44.066+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:44.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:44.184+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:44.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:44.184+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:44.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:44.184+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:44.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:44.186+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:45.060 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:44.800+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:45.060 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:44.800+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:45.060 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:44.800+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:45.060 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:44.801+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:45.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:45.056+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:45.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:45.056+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:45.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:45.056+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:45.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:45.057+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:45.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:45.142+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:45.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:45.142+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:45.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:45.142+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:45.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:45.143+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:46.077 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:45.768+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:46.077 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:45.768+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:46.077 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:45.768+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:46.078 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:45.768+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:46.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:46.073+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:46.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:46.073+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:46.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:46.073+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:46.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:46.074+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:46.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:46.185+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:46.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:46.185+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:46.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:46.185+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:46.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:46.186+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:46.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:18:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:18:46.628+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 17399 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:18:46.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:46.765+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:46.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:46.765+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:46.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:46.765+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:46.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:46.766+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:47.169 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:47.048+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:47.169 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:47.048+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:47.169 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:47.048+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:47.169 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:47.049+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:47.513 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:47.188+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:47.513 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:47.188+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:47.514 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:47.188+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:47.514 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:47.188+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:48.049 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:47.783+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:48.049 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:47.783+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:48.049 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:47.783+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:48.049 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:47.784+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:48.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:48.044+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:48.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:48.044+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:48.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:48.044+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:48.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:48.045+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:48.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:48.221+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:48.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:48.221+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:48.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:48.221+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:48.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:48.222+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:49.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:48.820+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:49.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:48.820+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:49.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:48.820+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:49.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:48.820+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:49.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:49.037+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:49.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:49.037+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:49.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:49.037+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:49.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:49.038+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:49.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:49.237+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:49.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:49.237+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:49.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:49.237+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:49.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:49.238+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:50.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:49.818+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:50.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:49.818+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:50.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:49.818+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:50.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:49.819+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:50.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:49.992+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:50.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:49.992+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:50.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:49.992+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:50.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:49.993+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:50.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:50.253+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:50.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:50.253+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:50.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:50.253+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:50.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:50.254+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:50.968 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:50.789+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:50.968 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:50.789+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:50.968 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:50.789+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:50.968 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:50.789+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:51.282 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:50.964+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:51.282 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:50.964+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:51.283 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:50.964+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:51.283 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:50.965+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:51.633 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:51.279+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:51.634 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:51.279+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:51.634 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:51.279+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:51.634 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:51.279+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:51.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:18:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:18:51.629+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 17481 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:18:51.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:51.770+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:51.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:51.770+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:51.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:51.770+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:51.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:51.771+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:52.297 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:51.919+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:52.298 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:51.919+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:52.298 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:51.919+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:52.298 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:51.921+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:52.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:52.293+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:52.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:52.293+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:52.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:52.293+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:52.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:52.294+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:53.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:52.889+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:53.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:52.889+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:53.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:52.889+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:53.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:52.890+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:53.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:52.736+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:53.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:52.736+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:53.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:52.736+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:53.160 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:52.737+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:53.596 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:53.338+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:53.596 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:53.338+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:53.597 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:53.338+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:53.597 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:53.339+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:53.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:53.730+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:53.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:53.730+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:53.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:53.730+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:53.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:53.731+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:54.362 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:53.933+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:54.363 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:53.933+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:54.363 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:53.933+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:54.363 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:53.934+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:54.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:54.359+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:54.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:54.359+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:54.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:54.359+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:54.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:54.360+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:54.963 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:54.700+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:54.963 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:54.700+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:54.963 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:54.700+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:54.963 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:54.701+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:55.368 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:54.959+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:55.369 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:54.959+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:55.369 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:54.959+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:55.369 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:54.960+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:55.660 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:55.365+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:55.661 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:55.365+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:55.661 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:55.365+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:55.661 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:55.365+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:55.970 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:55.655+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:55.970 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:55.655+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:55.970 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:55.655+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:55.970 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:55.656+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:56.379 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:55.966+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:56.379 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:55.966+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:56.379 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:55.966+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:56.380 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:55.967+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:56.635 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:56.375+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:56.636 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:56.375+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:56.636 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:56.375+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:56.636 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:56.376+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:56.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:18:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:18:56.631+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 17562 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:18:56.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:56.686+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:56.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:56.686+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:56.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:56.686+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:56.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:56.687+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:57.364 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:56.948+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:57.364 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:56.948+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:57.364 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:56.948+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:57.364 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:56.949+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:57.656 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:57.360+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:57.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:57.360+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:57.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:57.360+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:57.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:57.361+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:57.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:57.652+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:57.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:57.652+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:57.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:57.652+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:57.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:57.653+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:58.327 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:57.943+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:58.328 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:57.943+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:58.328 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:57.943+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:58.328 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:57.944+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:58.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:58.320+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:58.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:58.320+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:58.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:58.320+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:58.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:58.321+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:59.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:58.695+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:59.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:58.695+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:59.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:58.695+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:59.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:58.695+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:59.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:58.925+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:59.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:58.925+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:59.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:58.925+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:18:59.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:58.926+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:18:59.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:59.323+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:59.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:59.323+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:59.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:59.323+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:18:59.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:18:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:18:59.324+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:18:59.924 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:59.657+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:59.924 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:59.657+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:59.925 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:59.657+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:18:59.925 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:18:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:18:59.658+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:00.337 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:59.920+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:00.337 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:59.920+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:00.337 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:59.920+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:00.337 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:18:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:18:59.921+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:00.656 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:00.333+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:00.656 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:00.333+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:00.656 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:00.333+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:00.656 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:00.334+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:00.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:00.652+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:00.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:00.652+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:00.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:00.652+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:00.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:00.653+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:01.305 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:00.946+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:01.305 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:00.946+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:01.306 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:00.946+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:01.306 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:00.946+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:01.637 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:01.301+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:01.637 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:01.301+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:01.638 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:01.301+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:01.638 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:01.302+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:01.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:19:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:19:01.633+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 17643 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:19:01.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:01.666+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:01.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:01.666+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:01.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:01.666+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:01.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:01.666+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:02.304 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:01.932+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:02.305 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:01.932+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:02.305 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:01.932+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:02.305 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:01.932+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:02.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:02.301+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:02.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:02.301+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:02.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:02.301+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:02.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:02.302+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:02.978 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:02.658+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:02.978 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:02.658+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:02.978 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:02.658+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:02.979 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:02.658+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:03.313 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:02.974+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:03.314 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:02.974+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:03.314 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:02.974+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:03.314 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:02.975+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:03.597 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:03.309+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:03.597 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:03.310+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:03.597 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:03.310+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:03.597 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:03.311+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:03.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:03.615+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:03.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:03.615+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:03.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:03.615+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:03.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:03.616+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:04.364 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:03.967+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:04.365 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:03.967+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:04.365 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:03.967+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:04.365 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:03.968+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:04.644 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:04.360+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:04.644 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:04.360+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:04.644 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:04.360+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:04.644 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:04.361+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:04.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:04.639+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:04.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:04.639+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:04.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:04.639+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:04.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:04.640+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:05.406 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:04.999+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:05.406 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:04.999+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:05.406 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:04.999+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:05.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:05.000+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:05.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:05.639+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:05.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:05.639+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:05.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:05.639+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:05.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:05.639+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:05.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:05.401+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:05.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:05.401+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:05.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:05.401+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:05.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:05.402+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:06.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:05.954+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:06.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:05.954+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:06.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:05.954+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:06.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:05.955+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:06.763 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:19:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:19:06.634+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 17723 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:19:06.764 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:06.667+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:06.764 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:06.667+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:06.764 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:06.667+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:06.764 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:06.668+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:06.764 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:06.418+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:06.765 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:06.418+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:06.765 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:06.418+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:06.765 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:06.419+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:07.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:06.940+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:07.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:06.940+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:07.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:06.940+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:07.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:06.941+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:07.685 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:07.402+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:07.685 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:07.402+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:07.686 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:07.402+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:07.686 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:07.403+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:08.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:07.680+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:08.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:07.680+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:08.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:07.680+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:08.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:07.682+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:08.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:07.904+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:08.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:07.904+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:08.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:07.904+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:08.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:07.904+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:08.717 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:08.434+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:08.718 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:08.434+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:08.718 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:08.434+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:08.718 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:08.435+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:09.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:08.891+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:09.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:08.891+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:09.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:08.891+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:09.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:08.891+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:09.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:08.713+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:09.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:08.713+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:09.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:08.713+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:09.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:08.714+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:09.759 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:09.455+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:09.759 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:09.455+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:09.759 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:09.455+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:09.759 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:09.456+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:10.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:09.844+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:10.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:09.844+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:10.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:09.844+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:10.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:09.845+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:10.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:09.754+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:10.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:09.754+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:10.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:09.754+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:10.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:09.755+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:10.770 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:10.446+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:10.770 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:10.446+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:10.770 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:10.446+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:10.771 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:10.447+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:11.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:10.895+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:11.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:10.895+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:11.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:10.895+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:11.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:10.895+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:11.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:10.765+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:11.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:10.765+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:11.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:10.765+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:11.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:10.766+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:11.708 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:19:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:19:11.636+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 17805 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:19:11.708 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:11.427+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:11.708 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:11.427+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:11.709 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:11.427+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:11.709 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:11.429+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:12.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:11.789+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:12.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:11.789+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:12.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:11.789+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:12.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:11.789+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:12.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:11.919+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:12.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:11.919+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:12.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:11.919+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:12.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:11.921+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:12.812 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:12.440+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:12.812 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:12.440+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:12.813 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:12.440+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:12.813 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:12.441+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:13.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:12.808+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:13.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:12.808+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:13.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:12.808+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:13.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:12.809+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:13.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:12.935+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:13.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:12.935+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:13.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:12.935+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:13.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:12.936+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:13.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:13.396+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:13.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:13.396+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:13.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:13.396+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:13.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:13.397+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:14.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:13.784+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:14.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:13.784+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:14.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:13.784+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:14.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:13.785+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:14.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:13.946+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:14.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:13.946+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:14.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:13.946+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:14.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:13.947+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:14.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:14.359+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:14.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:14.359+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:14.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:14.359+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:14.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:14.360+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:15.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:14.782+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:15.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:14.782+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:15.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:14.782+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:15.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:14.783+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:15.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:14.978+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:15.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:14.978+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:15.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:14.978+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:15.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:14.979+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:15.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:15.315+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:15.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:15.315+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:15.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:15.315+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:15.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:15.316+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:16.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:15.829+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:16.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:15.829+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:16.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:15.829+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:16.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:15.830+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:16.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:15.977+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:16.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:15.977+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:16.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:15.977+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:16.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:15.979+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:16.642 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:16.301+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:16.643 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:16.301+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:16.643 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:16.301+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:16.643 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:16.301+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:16.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:19:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:19:16.638+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 17886 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:19:16.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:16.851+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:16.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:16.851+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:16.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:16.851+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:16.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:16.852+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:17.265 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:16.993+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:17.266 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:16.993+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:17.266 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:16.993+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:17.266 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:16.995+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:17.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:17.261+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:17.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:17.261+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:17.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:17.261+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:17.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:17.262+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:18.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:17.882+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:18.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:17.882+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:18.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:17.882+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:18.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:17.883+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:18.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:18.024+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:18.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:18.024+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:18.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:18.024+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:18.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:18.025+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:18.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:18.247+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:18.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:18.247+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:18.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:18.247+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:18.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:18.248+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:19.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:18.859+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:19.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:18.859+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:19.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:18.859+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:19.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:18.859+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:19.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:19.038+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:19.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:19.038+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:19.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:19.038+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:19.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:19.039+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:19.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:19.292+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:19.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:19.292+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:19.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:19.292+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:19.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:19.293+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:20.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:19.834+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:20.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:19.834+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:20.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:19.834+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:20.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:19.835+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:20.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:20.067+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:20.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:20.067+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:20.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:20.067+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:20.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:20.067+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:20.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:20.324+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:20.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:20.324+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:20.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:20.324+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:20.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:20.325+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:21.111 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:20.829+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:21.111 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:20.829+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:21.111 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:20.829+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:21.111 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:20.830+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:21.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:21.291+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:21.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:21.291+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:21.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:21.291+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:21.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:21.291+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:21.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:21.107+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:21.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:21.107+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:21.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:21.107+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:21.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:21.108+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:21.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:19:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:19:21.640+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 17968 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:19:21.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:21.844+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:21.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:21.844+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:21.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:21.844+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:21.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:21.845+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:22.099+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:22.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:22.099+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:22.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:22.099+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:22.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:22.100+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:22.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:22.269+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:22.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:22.269+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:22.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:22.269+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:22.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:22.270+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:23.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:22.869+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:23.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:22.870+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:23.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:22.870+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:23.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:22.871+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:23.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:23.105+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:23.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:23.105+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:23.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:23.105+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:48.463156+0000 (oldest deadline 2024-02-09T22:09:08.463156+0000) 2024-02-09T22:19:23.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:23.106+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:23.598 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:23.299+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:23.599 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:23.299+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:23.599 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:23.299+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:23.599 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:23.300+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:24.351 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:24.153+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:24.351 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:23.907+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:24.351 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:23.907+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:24.352 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:23.907+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:24.352 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:23.908+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:24.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:24.347+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:24.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:24.347+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:24.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:24.347+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:24.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:24.348+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:25.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:24.866+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:25.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:24.866+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:25.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:24.866+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:25.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:24.867+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:25.646 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:25.182+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:25.646 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:25.308+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:25.647 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:25.308+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:25.647 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:25.308+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:25.647 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:25.309+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:25.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:25.862+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:25.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:25.862+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:25.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:25.862+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:25.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:25.863+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:26.646 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:26.184+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:26.646 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:26.285+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:26.646 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:26.285+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:26.647 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:26.285+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:26.647 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:26.286+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:26.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:19:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:19:26.642+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 18047 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:19:26.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:26.834+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:26.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:26.834+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:26.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:26.834+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:26.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:26.835+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:27.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:27.219+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:27.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:27.295+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:27.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:27.295+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:27.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:27.295+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:27.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:27.296+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:28.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:27.792+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:28.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:27.792+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:28.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:27.792+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:08:51.218022+0000 (oldest deadline 2024-02-09T22:09:11.218022+0000) 2024-02-09T22:19:28.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:27.793+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:28.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:28.200+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:28.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:28.299+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:28.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:28.299+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:28.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:28.299+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:28.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:28.301+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:29.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:28.843+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:29.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:29.191+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:29.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:29.310+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:29.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:29.310+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:29.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:29.310+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:29.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:29.311+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:30.147 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:29.886+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:30.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:30.143+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:30.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:30.303+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:30.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:30.303+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:30.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:30.303+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:30.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:30.303+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:31.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:30.886+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:31.648 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:31.173+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:31.648 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:31.294+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:31.648 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:31.294+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:31.648 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:31.294+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:31.649 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:31.295+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:31.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:19:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:19:31.644+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 18128 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:19:31.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:31.871+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:32.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:32.221+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:32.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:32.251+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:32.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:32.251+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:32.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:32.251+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:32.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:32.252+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:33.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:32.877+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:33.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:33.201+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:33.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:33.255+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:33.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:33.255+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:33.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:33.255+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:33.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:33.256+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:34.250 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:33.927+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:34.251 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:34.169+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:34.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:34.246+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:34.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:34.246+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:34.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:34.246+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:34.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:34.247+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:35.142 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:34.912+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:35.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:35.138+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:35.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:35.273+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:35.273+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:35.273+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:35.274+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:36.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:36.098+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:36.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:35.876+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:36.650 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:36.247+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:36.650 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:36.247+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:36.650 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:36.247+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:36.650 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:36.247+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:36.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:19:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:19:36.646+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 18208 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:19:36.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:36.845+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:37.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:37.056+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:37.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:37.280+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:37.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:37.280+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:37.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:37.280+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:37.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:37.280+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:38.106 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:37.811+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:38.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:38.102+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:38.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:38.231+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:38.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:38.231+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:38.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:38.231+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:38.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:38.232+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:39.114 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:38.806+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:39.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:39.111+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:39.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:39.225+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:39.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:39.225+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:39.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:39.225+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:09:03.098727+0000 (oldest deadline 2024-02-09T22:09:23.098727+0000) 2024-02-09T22:19:39.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:39.226+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:40.136 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:39.767+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:40.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:40.185+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:40.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:40.132+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:41.125 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:40.767+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:41.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:41.216+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:41.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:41.122+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:41.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:19:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:19:41.647+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 18289 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:19:41.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:41.763+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:42.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:42.225+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:42.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:42.076+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:43.072 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:42.719+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:43.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:43.251+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:43.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:43.069+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:43.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:43.714+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:44.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:44.228+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:44.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:44.117+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:45.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:44.763+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:45.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:45.233+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:45.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:45.153+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:45.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:45.153+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:45.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:45.153+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:45.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:45.154+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:46.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:45.783+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:46.653 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:46.253+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:46.654 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:46.175+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:46.654 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:46.175+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:46.654 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:46.175+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:46.654 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:46.177+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:46.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:19:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:19:46.649+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 18369 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:19:46.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:46.778+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:47.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:47.205+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:47.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:47.193+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:47.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:47.193+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:47.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:47.193+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:47.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:47.194+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:48.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:47.743+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:48.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:48.246+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:48.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:48.154+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:48.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:48.154+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:48.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:48.154+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:48.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:48.154+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:49.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:48.694+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:49.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:49.214+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:49.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:49.184+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:49.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:49.184+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:49.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:49.184+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:49.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:49.185+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:50.150 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:49.679+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:50.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:50.146+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:50.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:50.146+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:50.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:50.146+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:50.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:50.147+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:50.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:50.225+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:51.119 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:50.705+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:19:51.120 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:50.705+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:19:51.120 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:50.705+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:19:51.120 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:50.706+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:51.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:51.115+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:51.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:51.115+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:51.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:51.115+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:51.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:51.116+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:51.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:51.195+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:51.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:19:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:19:51.651+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 18451 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:19:51.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:51.661+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:19:51.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:51.661+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:19:51.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:51.661+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:19:51.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:51.662+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:52.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:52.212+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:52.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:52.114+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:52.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:52.114+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:52.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:52.114+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:52.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:52.115+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:52.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:52.640+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:19:52.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:52.640+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:19:52.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:52.640+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:19:52.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:52.641+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:53.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:53.241+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:53.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:53.068+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:53.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:53.068+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:53.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:53.068+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:53.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:53.068+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:53.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:53.606+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:19:53.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:53.606+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:19:53.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:53.606+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:19:53.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:53.607+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:54.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:54.286+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:54.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:54.037+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:54.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:54.037+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:54.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:54.037+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:54.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:54.038+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:54.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:54.568+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:19:54.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:54.568+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:19:54.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:54.568+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:19:54.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:54.570+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:55.271 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:55.013+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:55.271 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:55.013+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:55.271 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:55.013+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:55.271 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:55.014+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:55.575 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:55.267+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:55.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:55.570+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:19:55.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:55.570+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:19:55.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:55.570+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:19:55.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:55.570+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:56.292 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:55.988+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:56.292 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:55.988+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:56.292 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:55.988+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:56.292 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:55.989+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:56.582 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:56.288+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:56.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:19:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:19:56.653+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 18532 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:19:56.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:56.578+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:19:56.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:56.578+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:19:56.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:56.578+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:19:56.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:56.580+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:57.280 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:57.012+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:57.281 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:57.012+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:57.281 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:57.012+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:57.281 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:57.013+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:57.614 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:57.277+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:57.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:57.610+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:19:57.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:57.610+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:19:57.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:57.610+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:19:57.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:57.612+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:58.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:58.258+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:58.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:58.054+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:58.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:58.054+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:58.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:58.054+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:58.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:58.055+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:58.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:58.635+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:19:58.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:58.635+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:19:58.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:58.635+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:19:58.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:58.636+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:59.283 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:59.007+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:59.284 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:59.007+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:59.284 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:59.007+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:19:59.284 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:19:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:19:59.008+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:19:59.643 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:19:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:19:59.280+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:19:59.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:59.640+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:19:59.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:59.640+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:19:59.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:59.640+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:19:59.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:19:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:19:59.640+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:00.294 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:00.001+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:00.295 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:00.001+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:00.295 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:00.001+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:00.295 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:00.002+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:00.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:00.290+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:00.972 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:00.663+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:00.972 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:00.663+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:00.972 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:00.663+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:00.972 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:00.664+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:01.337 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:00.968+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:01.337 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:00.968+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:01.338 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:00.968+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:01.338 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:00.969+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:01.643 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:01.333+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:01.644 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:01.333+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:01.644 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:01.333+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:01.644 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:01.334+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:01.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:20:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:20:01.655+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 18614 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:20:01.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:01.639+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:01.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:01.639+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:01.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:01.639+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:01.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:01.640+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:02.356 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:01.929+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:02.356 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:01.929+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:02.357 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:01.929+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:02.357 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:01.930+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:02.652 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:02.352+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:02.652 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:02.352+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:02.652 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:02.352+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:02.655 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:02.353+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:02.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:02.647+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:02.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:02.647+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:02.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:02.647+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:02.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:02.649+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:03.400 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:02.909+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:03.400 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:02.909+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:03.400 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:02.909+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:03.400 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:02.910+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:03.651 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:03.396+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:03.651 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:03.396+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:03.651 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:03.396+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:03.651 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:03.397+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:03.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:03.647+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:03.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:03.647+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:03.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:03.647+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:03.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:03.648+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:04.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:03.908+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:04.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:03.908+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:04.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:03.908+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:04.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:03.908+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:04.885 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:04.616+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:04.885 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:04.616+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:04.885 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:04.616+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:04.885 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:04.617+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:04.886 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:04.405+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:04.886 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:04.405+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:04.886 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:04.405+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:04.886 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:04.406+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:05.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:04.881+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:05.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:04.881+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:05.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:04.881+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:05.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:04.882+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:05.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:05.586+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:05.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:05.586+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:05.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:05.586+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:05.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:05.587+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:05.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:05.449+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:05.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:05.449+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:05.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:05.449+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:05.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:05.450+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:06.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:05.931+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:06.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:05.931+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:06.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:05.931+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:06.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:05.932+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:06.684 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:20:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:20:06.657+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 18694 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:20:06.684 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:06.604+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:06.684 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:06.604+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:06.685 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:06.604+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:06.685 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:06.605+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:06.685 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:06.425+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:06.685 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:06.425+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:06.685 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:06.425+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:06.685 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:06.426+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:07.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:06.951+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:07.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:06.951+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:07.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:06.951+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:07.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:06.952+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:07.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:07.648+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:07.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:07.648+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:07.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:07.648+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:07.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:07.649+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:07.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:07.421+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:07.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:07.421+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:07.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:07.421+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:07.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:07.422+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:08.391 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:07.955+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:08.392 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:07.955+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:08.392 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:07.955+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:08.392 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:07.956+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:08.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:08.388+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:08.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:08.388+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:08.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:08.388+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:08.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:08.388+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:09.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:08.933+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:09.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:08.933+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:09.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:08.933+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:09.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:08.934+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:09.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:08.691+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:09.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:08.691+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:09.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:08.691+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:09.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:08.692+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:09.721 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:09.409+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:09.722 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:09.409+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:09.722 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:09.409+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:09.722 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:09.410+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:09.980 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:09.718+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:09.981 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:09.718+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:09.981 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:09.718+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:09.981 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:09.719+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:10.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:09.977+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:10.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:09.977+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:10.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:09.977+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:10.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:09.978+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:10.694 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:10.422+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:10.694 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:10.422+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:10.695 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:10.422+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:10.695 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:10.423+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:11.014 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:10.690+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:11.689 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:10.690+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:11.689 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:10.690+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:11.689 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:10.691+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:11.690 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:11.010+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:11.691 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:11.010+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:11.691 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:11.010+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:11.691 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:11.011+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:11.711 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:20:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:20:11.658+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 18776 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:20:11.711 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:11.447+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:11.712 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:11.447+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:11.712 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:11.447+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:11.712 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:11.448+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:11.987 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:11.732+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:11.988 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:11.732+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:11.988 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:11.732+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:11.988 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:11.733+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:12.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:11.983+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:12.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:11.983+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:12.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:11.983+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:12.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:11.985+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:12.774 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:12.496+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:12.774 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:12.496+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:12.774 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:12.496+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:12.774 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:12.498+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:13.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:12.770+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:13.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:12.770+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:13.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:12.770+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:13.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:12.771+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:13.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:13.013+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:13.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:13.013+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:13.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:13.013+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:13.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:13.014+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:13.795 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:13.464+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:13.795 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:13.464+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:13.796 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:13.464+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:13.796 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:13.465+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:14.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:13.791+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:14.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:13.791+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:14.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:13.791+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:14.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:13.792+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:14.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:14.026+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:14.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:14.026+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:14.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:14.026+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:14.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:14.027+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:14.781 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:14.472+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:14.781 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:14.472+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:14.781 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:14.472+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:14.782 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:14.473+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:15.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:14.999+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:15.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:14.999+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:15.161 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:14.999+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:15.161 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:15.000+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:15.161 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:14.777+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:15.161 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:14.777+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:15.161 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:14.777+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:15.161 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:14.778+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:15.772 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:15.451+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:15.773 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:15.451+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:15.773 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:15.451+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:15.773 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:15.452+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:16.039 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:15.768+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:16.039 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:15.768+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:16.039 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:15.768+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:16.039 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:15.769+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:16.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:16.035+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:16.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:16.035+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:16.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:16.035+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:16.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:16.036+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:16.790 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:20:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:20:16.660+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 18859 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:20:16.791 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:16.468+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:16.791 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:16.468+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:16.791 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:16.468+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:16.791 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:16.469+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:17.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:17.005+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:17.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:17.005+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:17.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:17.005+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:17.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:17.006+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:17.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:16.799+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:17.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:16.799+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:17.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:16.799+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:17.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:16.800+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:17.819 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:17.438+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:17.819 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:17.438+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:17.820 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:17.438+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:17.820 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:17.440+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:18.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:18.037+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:18.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:18.037+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:18.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:18.037+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:18.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:18.038+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:18.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:17.815+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:18.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:17.815+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:18.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:17.815+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:18.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:17.816+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:18.863 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:18.478+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:18.864 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:18.478+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:18.864 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:18.478+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:18.864 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:18.479+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:19.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:19.067+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:19.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:19.067+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:19.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:19.067+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:19.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:19.068+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:19.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:18.859+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:19.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:18.860+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:19.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:18.860+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:19.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:18.861+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:19.842 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:19.480+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:19.843 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:19.480+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:19.843 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:19.480+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:19.843 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:19.481+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:20.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:19.839+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:20.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:19.839+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:20.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:19.839+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:20.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:19.840+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:20.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:20.050+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:20.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:20.050+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:20.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:20.050+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:20.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:20.051+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:20.856 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:20.519+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:20.856 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:20.519+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:20.857 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:20.519+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:20.857 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:20.520+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:21.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:21.034+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:21.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:21.034+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:21.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:21.034+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:21.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:21.035+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:21.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:20.852+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:21.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:20.852+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:21.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:20.852+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:21.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:20.854+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:21.788 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:20:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:20:21.662+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 18943 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:20:21.789 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:21.474+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:21.789 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:21.474+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:21.789 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:21.474+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:21.789 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:21.475+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:22.054 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:21.865+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:22.054 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:21.865+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:22.055 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:21.865+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:22.055 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:21.866+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:22.050+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:22.050+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:22.050+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:22.051+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:22.846 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:22.448+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:22.846 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:22.448+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:22.846 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:22.448+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:22.846 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:22.448+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:23.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:22.842+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:23.160 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:22.842+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:23.160 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:22.842+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:23.160 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:22.843+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:23.161 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:23.045+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:23.161 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:23.045+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:23.161 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:23.045+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:23.161 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:23.046+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:23.844 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:23.460+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:23.844 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:23.460+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:23.844 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:23.460+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:23.844 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:23.460+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:24.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:24.010+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:24.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:24.010+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:24.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:24.010+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:24.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:24.012+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:24.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:23.840+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:24.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:23.840+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:24.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:23.840+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:24.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:23.841+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:24.864 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:24.453+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:24.864 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:24.453+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:24.865 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:24.453+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:24.865 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:24.454+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:25.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:24.860+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:25.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:24.860+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:25.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:24.860+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:25.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:24.861+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:25.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:24.965+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:25.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:24.965+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:25.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:24.965+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:25.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:24.965+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:25.897 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:25.500+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:25.897 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:25.500+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:25.897 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:25.501+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:25.897 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:25.501+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:26.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:25.950+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:26.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:25.950+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:26.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:25.950+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:26.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:25.951+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:26.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:25.893+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:26.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:25.893+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:26.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:25.893+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:26.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:25.894+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:26.793 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:20:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:20:26.664+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 19026 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:20:26.793 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:26.459+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:26.793 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:26.459+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:26.794 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:26.459+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:26.794 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:26.460+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:27.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:26.972+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:27.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:26.972+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:27.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:26.972+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:27.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:26.973+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:27.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:26.857+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:27.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:26.857+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:27.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:26.857+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:27.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:26.857+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:27.895 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:27.474+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:27.896 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:27.474+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:27.896 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:27.474+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:27.896 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:27.475+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:28.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:27.892+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:28.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:27.892+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:28.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:27.892+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:28.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:27.893+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:28.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:27.932+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:28.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:27.932+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:28.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:27.932+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:28.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:27.933+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:28.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:28.462+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:28.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:28.462+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:28.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:28.462+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:28.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:28.463+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:29.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:28.925+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:29.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:28.925+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:29.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:28.925+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:29.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:28.926+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:29.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:28.940+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:29.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:28.940+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:29.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:28.940+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:29.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:28.941+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:29.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:29.451+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:29.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:29.451+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:29.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:29.451+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:29.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:29.452+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:30.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:29.966+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:30.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:29.966+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:30.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:29.966+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:30.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:29.966+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:30.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:29.958+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:30.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:29.958+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:30.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:29.958+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:30.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:29.958+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:30.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:30.461+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:30.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:30.461+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:30.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:30.461+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:30.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:30.462+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:31.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:30.963+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:31.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:30.963+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:31.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:30.963+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:31.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:30.963+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:31.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:30.929+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:31.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:30.929+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:31.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:30.929+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:31.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:30.930+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:31.794 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:20:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:20:31.666+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 19110 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:20:31.795 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:31.464+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:31.795 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:31.464+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:31.795 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:31.464+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:31.795 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:31.465+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:32.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:31.883+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:32.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:31.883+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:32.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:31.883+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:32.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:31.883+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:32.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:31.977+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:32.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:31.977+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:32.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:31.977+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:32.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:31.978+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:32.903 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:32.425+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:32.903 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:32.425+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:32.903 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:32.425+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:32.903 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:32.426+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:33.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:33.004+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:33.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:33.004+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:33.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:33.004+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:33.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:33.005+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:33.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:32.899+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:33.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:32.899+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:33.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:32.899+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:33.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:32.900+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:33.498 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:33.441+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:33.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:33.441+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:33.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:33.441+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:33.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:33.441+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:34.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:33.946+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:34.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:33.946+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:34.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:33.946+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:34.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:33.947+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:34.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:33.972+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:34.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:33.972+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:34.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:33.972+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:34.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:33.973+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:34.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:34.471+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:34.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:34.471+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:34.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:34.471+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:34.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:34.471+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:35.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:35.023+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:35.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:35.023+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:35.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:35.023+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:35.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:35.024+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:35.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:34.940+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:35.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:34.940+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:35.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:34.940+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:35.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:34.941+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:35.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:35.434+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:35.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:35.434+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:35.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:35.434+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:35.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:35.435+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:36.302 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:36.031+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:36.302 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:36.031+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:36.302 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:36.031+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:36.302 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:36.032+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:36.303 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:35.955+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:36.303 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:35.955+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:36.303 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:35.955+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:36.303 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:35.956+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:36.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:36.389+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:36.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:36.389+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:36.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:36.389+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:36.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:36.390+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:36.958 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:20:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:20:36.668+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 19193 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:20:37.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:36.995+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:37.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:36.995+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:37.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:36.995+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:37.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:36.996+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:37.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:36.954+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:37.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:36.954+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:37.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:36.954+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:37.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:36.955+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:37.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:37.433+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:37.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:37.433+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:37.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:37.433+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:37.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:37.434+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:38.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:38.036+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:38.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:38.036+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:38.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:38.036+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:38.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:38.038+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:38.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:37.923+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:38.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:37.923+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:38.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:37.923+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:38.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:37.923+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:38.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:38.449+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:38.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:38.449+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:38.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:38.449+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:38.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:38.450+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:39.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:39.029+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:39.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:39.029+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:39.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:39.029+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:39.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:39.031+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:39.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:38.922+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:39.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:38.922+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:39.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:38.922+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:39.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:38.922+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:39.898 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:39.487+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:39.898 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:39.487+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:39.898 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:39.487+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:39.898 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:39.488+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:40.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:40.062+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:40.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:40.062+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:40.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:40.062+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:40.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:40.063+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:40.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:39.894+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:40.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:39.894+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:40.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:39.894+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:40.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:39.895+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:40.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:40.486+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:40.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:40.486+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:40.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:40.486+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:40.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:40.487+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:41.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:41.038+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:41.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:41.038+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:41.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:41.038+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:41.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:41.039+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:41.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:40.943+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:41.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:40.943+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:41.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:40.943+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:41.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:40.944+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:41.797 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:20:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:20:41.669+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 19277 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:20:41.798 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:41.508+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:41.798 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:41.508+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:41.798 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:41.508+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:41.798 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:41.509+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:42.088 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:41.983+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:42.088 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:41.983+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:42.088 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:41.983+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:42.088 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:41.984+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:42.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:42.083+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:42.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:42.083+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:42.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:42.083+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:42.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:42.084+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:42.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:42.492+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:42.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:42.492+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:42.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:42.492+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:42.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:42.493+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:43.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:43.058+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:43.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:43.058+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:43.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:43.058+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:43.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:43.059+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:43.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:43.026+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:43.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:43.026+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:43.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:43.026+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:43.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:43.027+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:43.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:43.537+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:43.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:43.537+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:43.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:43.537+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:43.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:43.538+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:44.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:44.064+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:44.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:44.064+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:44.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:44.064+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:44.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:44.065+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:44.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:44.063+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:44.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:44.063+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:44.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:44.063+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:44.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:44.064+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:44.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:44.509+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:44.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:44.509+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:44.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:44.509+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:44.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:44.510+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:45.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:45.086+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:45.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:45.086+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:45.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:45.086+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:45.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:45.087+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:45.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:45.030+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:45.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:45.030+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:45.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:45.030+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:45.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:45.031+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:45.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:45.534+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:45.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:45.534+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:45.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:45.534+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:45.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:45.535+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:46.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:45.987+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:46.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:45.987+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:46.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:45.987+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:46.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:45.988+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:46.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:46.069+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:46.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:46.069+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:46.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:46.069+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:46.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:46.069+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:46.800 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:20:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:20:46.670+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 19360 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:20:46.800 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:46.535+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:46.801 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:46.535+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:46.801 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:46.535+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:46.801 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:46.536+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:47.124 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:47.036+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:47.124 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:47.036+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:47.124 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:47.036+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:47.124 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:47.036+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:47.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:47.120+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:47.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:47.120+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:47.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:47.120+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:47.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:47.120+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:47.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:47.530+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:47.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:47.530+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:47.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:47.530+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:47.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:47.530+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:48.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:48.120+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:48.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:48.120+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:48.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:48.120+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:48.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:48.120+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:48.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:48.067+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:48.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:48.067+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:48.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:48.067+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:48.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:48.068+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:48.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:48.498+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:48.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:48.498+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:48.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:48.498+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:48.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:48.498+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:49.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:49.109+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:49.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:49.109+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:49.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:49.109+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:49.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:49.110+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:49.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:49.044+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:49.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:49.044+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:49.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:49.044+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:49.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:49.045+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:49.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:49.514+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:49.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:49.514+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:49.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:49.514+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:49.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:49.515+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:50.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:50.131+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:50.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:50.131+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:50.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:50.131+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:50.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:50.133+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:50.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:49.998+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:50.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:49.998+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:50.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:49.998+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:50.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:49.999+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:50.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:50.551+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:50.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:50.551+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:50.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:50.551+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:50.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:50.552+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:51.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:50.972+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:51.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:50.972+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:51.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:50.972+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:51.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:50.973+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:51.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:51.084+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:51.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:51.084+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:51.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:51.084+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:51.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:51.085+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:51.799 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:20:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:20:51.673+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 19444 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:20:51.799 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:51.529+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:51.800 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:51.529+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:51.800 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:51.529+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:51.800 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:51.529+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:52.115 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:52.020+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:52.116 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:52.020+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:52.116 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:52.020+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:52.116 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:52.021+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:52.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:52.112+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:52.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:52.112+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:52.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:52.112+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:52.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:52.113+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:52.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:52.487+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:52.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:52.487+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:52.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:52.487+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:52.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:52.488+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:53.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:53.108+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:53.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:53.108+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:53.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:53.108+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:53.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:53.110+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:53.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:53.019+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:53.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:53.019+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:53.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:53.019+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:53.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:53.020+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:53.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:53.532+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:53.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:53.532+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:53.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:53.532+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:53.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:53.533+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:54.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:54.130+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:54.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:54.130+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:54.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:54.130+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:54.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:54.131+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:54.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:53.983+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:54.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:53.983+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:54.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:53.983+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:54.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:53.983+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:54.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:54.544+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:54.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:54.544+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:54.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:54.544+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:54.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:54.545+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:55.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:55.022+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:55.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:55.022+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:55.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:55.022+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:55.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:55.023+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:55.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:55.112+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:55.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:55.112+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:55.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:55.112+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:55.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:55.113+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:55.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:55.584+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:55.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:55.584+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:55.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:55.584+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:55.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:55.585+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:56.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:56.126+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:56.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:56.126+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:56.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:56.126+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:56.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:56.126+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:56.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:56.017+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:56.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:56.017+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:56.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:56.017+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:56.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:56.017+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:56.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:20:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:20:56.675+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 19526 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:20:56.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:56.614+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:56.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:56.614+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:56.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:56.614+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:56.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:56.615+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:57.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:57.094+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:57.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:57.094+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:57.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:57.094+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:57.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:57.095+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:57.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:57.051+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:57.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:57.051+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:57.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:57.051+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:57.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:57.052+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:58.011 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:57.664+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:58.011 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:57.664+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:58.011 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:57.664+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:58.012 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:57.665+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:58.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:58.052+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:58.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:58.052+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:58.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:58.052+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:58.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:58.053+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:58.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:58.007+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:58.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:58.007+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:58.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:58.007+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:58.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:58.008+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:20:58.975 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:58.688+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:58.976 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:58.688+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:58.976 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:58.688+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:20:58.976 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:58.689+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:59.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:59.035+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:59.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:59.035+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:59.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:59.035+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:20:59.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:20:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:20:59.036+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:20:59.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:58.971+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:59.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:58.971+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:59.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:58.971+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:20:59.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:20:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:20:58.973+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:00.004 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:59.654+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:00.005 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:59.654+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:00.005 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:59.654+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:00.005 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:20:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:20:59.655+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:00.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:00.001+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:00.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:00.001+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:00.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:00.001+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:00.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:00.002+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:00.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:00.000+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:00.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:00.000+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:00.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:00.000+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:00.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:00.001+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:00.878 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:00.610+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:00.878 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:00.610+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:00.878 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:00.610+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:00.878 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:00.611+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:01.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:00.985+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:01.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:00.985+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:01.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:00.985+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:01.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:00.986+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:01.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:00.967+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:01.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:00.967+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:01.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:00.967+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:01.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:00.968+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:01.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:21:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:21:01.677+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 19609 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:21:01.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:01.623+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:01.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:01.623+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:01.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:01.623+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:01.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:01.624+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:02.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:01.966+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:02.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:01.966+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:02.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:01.966+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:02.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:01.967+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:02.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:01.931+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:02.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:01.931+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:02.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:01.931+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:02.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:01.932+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:02.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:02.584+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:02.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:02.584+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:02.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:02.584+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:02.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:02.585+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:03.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:02.921+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:03.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:02.921+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:03.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:02.921+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:03.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:02.922+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:03.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:02.929+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:03.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:02.929+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:03.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:02.929+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:03.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:02.929+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:03.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:03.542+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:03.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:03.542+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:03.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:03.542+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:03.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:03.543+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:04.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:03.900+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:04.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:03.900+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:04.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:03.900+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:04.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:03.901+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:04.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:03.908+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:04.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:03.908+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:04.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:03.908+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:04.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:03.908+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:04.887 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:04.549+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:04.887 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:04.549+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:04.887 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:04.549+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:04.887 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:04.549+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:05.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:04.951+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:05.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:04.951+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:05.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:04.951+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:05.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:04.952+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:05.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:04.883+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:05.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:04.883+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:05.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:04.883+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:05.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:04.884+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:05.890 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:05.590+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:05.890 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:05.590+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:05.890 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:05.590+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:05.890 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:05.590+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:06.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:06.000+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:06.160 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:06.000+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:06.160 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:06.000+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:06.160 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:06.001+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:06.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:05.886+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:06.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:05.886+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:06.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:05.886+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:06.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:05.887+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:06.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:21:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:21:06.679+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 19691 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:21:06.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:06.586+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:06.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:06.586+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:06.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:06.586+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:06.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:06.587+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:07.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:06.993+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:07.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:06.993+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:07.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:06.993+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:07.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:06.994+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:07.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:06.936+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:07.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:06.936+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:07.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:06.936+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:07.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:06.937+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:07.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:07.582+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:07.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:07.582+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:07.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:07.582+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:07.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:07.583+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:08.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:07.965+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:08.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:07.965+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:08.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:07.965+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:08.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:07.966+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:08.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:07.983+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:08.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:07.983+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:08.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:07.983+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:08.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:07.983+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:08.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:08.603+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:08.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:08.603+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:08.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:08.603+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:08.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:08.604+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:09.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:08.971+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:09.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:08.971+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:09.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:08.971+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:09.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:08.972+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:09.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:09.021+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:09.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:09.021+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:09.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:09.021+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:09.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:09.022+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:09.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:09.592+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:09.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:09.592+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:09.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:09.592+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:09.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:09.594+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:10.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:09.938+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:10.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:09.938+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:10.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:09.938+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:10.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:09.938+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:10.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:09.985+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:10.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:09.985+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:10.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:09.985+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:10.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:09.986+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:10.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:10.624+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:10.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:10.624+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:10.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:10.624+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:10.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:10.625+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:11.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:10.944+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:11.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:10.944+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:11.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:10.944+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:11.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:10.944+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:11.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:10.929+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:11.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:10.929+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:11.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:10.929+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:11.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:10.930+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:11.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:21:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:21:11.681+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 19775 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:21:11.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:11.584+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:11.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:11.584+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:11.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:11.584+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:11.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:11.586+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:12.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:11.950+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:12.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:11.950+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:12.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:11.950+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:12.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:11.950+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:12.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:11.933+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:12.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:11.933+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:12.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:11.933+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:12.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:11.934+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:12.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:12.591+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:12.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:12.591+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:12.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:12.591+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:12.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:12.592+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:13.166 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:12.930+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:13.166 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:12.930+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:13.166 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:12.930+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:13.166 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:12.930+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:13.166 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:12.905+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:13.167 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:12.905+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:13.167 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:12.905+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:13.167 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:12.906+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:13.866 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:13.551+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:13.866 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:13.551+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:13.866 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:13.551+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:13.867 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:13.552+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:14.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:13.862+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:14.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:13.862+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:14.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:13.862+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:14.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:13.862+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:14.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:13.882+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:14.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:13.882+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:14.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:13.882+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:14.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:13.883+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:14.843 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:14.565+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:14.843 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:14.565+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:14.843 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:14.565+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:14.843 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:14.566+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:15.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:14.839+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:15.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:14.839+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:15.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:14.839+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:15.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:14.840+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:15.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:14.882+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:15.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:14.882+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:15.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:14.882+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:15.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:14.882+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:15.851 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:15.811+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:15.851 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:15.811+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:15.851 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:15.811+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:15.852 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:15.812+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:15.852 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:15.564+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:15.852 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:15.564+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:15.852 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:15.564+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:15.852 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:15.565+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:16.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:15.847+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:16.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:15.847+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:16.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:15.847+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:16.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:15.848+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:16.831 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:21:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:21:16.683+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 19857 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:21:16.832 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:16.577+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:16.832 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:16.577+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:16.832 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:16.577+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:16.832 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:16.578+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:17.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:16.828+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:17.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:16.828+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:17.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:16.828+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:17.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:16.829+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:17.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:16.857+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:17.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:16.857+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:17.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:16.857+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:17.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:16.858+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:17.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:17.604+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:17.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:17.604+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:17.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:17.604+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:17.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:17.605+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:17.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:17.790+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:17.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:17.790+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:17.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:17.790+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:17.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:17.791+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:17.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:17.834+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:17.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:17.834+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:17.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:17.834+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:17.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:17.835+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:18.840 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:18.792+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:18.840 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:18.792+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:18.840 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:18.792+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:18.840 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:18.792+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:18.840 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:18.572+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:18.841 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:18.572+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:18.841 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:18.572+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:18.841 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:18.573+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:19.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:18.836+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:19.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:18.836+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:19.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:18.836+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:19.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:18.837+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:19.880 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:19.551+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:19.880 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:19.551+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:19.880 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:19.551+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:19.880 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:19.552+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:19.881 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:19.758+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:19.881 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:19.758+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:19.881 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:19.758+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:19.881 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:19.758+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:20.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:19.876+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:20.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:19.876+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:20.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:19.876+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:20.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:19.877+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:20.870 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:20.760+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:20.870 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:20.760+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:20.871 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:20.760+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:20.871 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:20.760+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:20.871 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:20.535+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:20.871 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:20.535+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:20.871 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:20.535+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:20.871 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:20.536+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:21.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:20.866+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:21.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:20.866+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:21.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:20.866+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:21.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:20.867+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:21.794 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:21:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:21:21.685+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 19941 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:21:21.794 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:21.492+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:21.795 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:21.492+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:21.795 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:21.492+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:21.795 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:21.493+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:22.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:21.890+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:22.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:21.890+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:22.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:21.890+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:22.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:21.891+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:22.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:21.790+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:22.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:21.790+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:22.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:21.790+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:22.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:21.791+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:22.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:22.535+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:22.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:22.535+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:22.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:22.535+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:22.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:22.536+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:23.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:22.758+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:23.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:22.758+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:23.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:22.758+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:23.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:22.759+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:23.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:22.900+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:23.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:22.900+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:23.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:22.900+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:23.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:22.901+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:23.800 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:23.542+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:23.801 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:23.542+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:23.801 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:23.542+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:23.801 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:23.543+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:24.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:23.906+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:24.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:23.906+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:24.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:23.906+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:24.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:23.907+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:24.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:23.797+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:24.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:23.797+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:24.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:23.797+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:24.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:23.798+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:24.781 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:24.508+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:24.782 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:24.508+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:24.782 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:24.508+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:24.782 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:24.508+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:25.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:24.778+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:25.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:24.778+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:25.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:24.778+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:25.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:24.779+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:25.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:24.910+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:25.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:24.911+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:25.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:24.911+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:25.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:24.911+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:25.824 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:25.484+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:25.824 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:25.484+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:25.824 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:25.484+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:25.824 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:25.485+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:26.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:25.820+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:26.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:25.820+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:26.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:25.820+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:26.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:25.821+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:26.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:25.938+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:26.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:25.938+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:26.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:25.938+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:26.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:25.939+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:26.714 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:21:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:21:26.687+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 20021 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:21:26.715 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:26.454+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:26.715 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:26.454+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:26.715 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:26.454+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:26.715 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:26.455+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:26.967 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:26.834+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:26.967 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:26.834+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:26.967 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:26.834+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:26.967 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:26.835+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:27.280 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:26.963+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:27.280 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:26.963+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:27.280 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:26.963+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:27.280 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:26.964+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:27.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:27.477+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:27.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:27.477+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:27.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:27.477+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:27.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:27.478+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:28.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:27.827+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:28.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:27.827+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:28.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:27.828+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:28.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:27.828+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:28.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:28.013+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:28.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:28.013+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:28.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:28.013+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:28.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:28.014+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:28.832 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:28.464+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:28.832 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:28.464+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:28.832 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:28.464+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:28.833 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:28.464+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:29.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:29.014+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:29.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:29.014+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:29.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:29.014+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:29.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:29.015+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:29.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:28.828+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:29.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:28.828+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:29.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:28.828+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:29.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:28.829+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:29.878 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:29.491+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:29.878 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:29.491+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:29.878 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:29.491+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:29.878 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:29.492+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:30.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:29.874+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:30.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:29.874+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:30.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:29.874+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:30.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:29.875+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:30.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:30.052+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:30.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:30.052+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:30.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:30.052+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:30.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:30.053+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:30.879 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:30.467+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:30.879 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:30.467+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:30.879 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:30.467+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:30.879 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:30.468+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:31.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:31.078+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:31.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:31.078+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:31.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:31.078+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:31.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:31.079+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:31.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:30.875+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:31.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:30.875+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:31.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:30.875+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:31.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:30.875+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:31.716 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:21:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:21:31.689+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 20099 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:21:31.717 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:31.453+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:31.717 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:31.453+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:31.717 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:31.453+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:31.717 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:31.454+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:32.114 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:31.909+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:32.114 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:31.909+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:32.114 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:31.909+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:32.114 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:31.910+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:32.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:32.110+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:32.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:32.110+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:32.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:32.110+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:32.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:32.110+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:32.883 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:32.493+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:32.883 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:32.493+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:32.883 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:32.493+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:32.883 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:32.494+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:33.155 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:32.879+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:33.156 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:32.879+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:33.156 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:32.879+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:33.156 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:32.880+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:33.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:33.152+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:33.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:33.152+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:33.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:33.152+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:33.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:33.153+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:33.886 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:33.477+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:33.886 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:33.477+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:33.887 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:33.477+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:33.887 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:33.478+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:34.156 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:33.882+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:34.156 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:33.882+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:34.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:33.882+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:34.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:33.883+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:34.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:34.152+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:34.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:34.152+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:34.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:34.152+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:34.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:34.153+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:34.888 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:34.471+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:34.888 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:34.471+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:34.889 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:34.471+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:34.889 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:34.472+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:35.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:34.884+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:35.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:34.884+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:35.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:34.884+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:35.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:34.885+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:35.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:35.118+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:35.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:35.118+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:35.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:35.118+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:35.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:35.119+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:35.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:35.453+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:35.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:35.453+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:35.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:35.454+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:35.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:35.454+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:36.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:35.910+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:36.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:35.910+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:36.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:35.910+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:36.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:35.911+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:36.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:36.140+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:36.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:36.140+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:36.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:36.140+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:36.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:36.141+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:36.718 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:21:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:21:36.691+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 20176 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:21:36.718 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:36.453+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:36.719 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:36.453+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:36.719 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:36.453+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:36.719 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:36.454+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:37.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:36.941+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:37.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:36.941+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:37.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:36.941+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:37.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:36.942+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:37.450 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:37.156+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:37.451 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:37.156+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:37.451 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:37.156+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:37.451 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:37.157+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:37.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:37.446+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:37.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:37.446+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:37.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:37.446+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:37.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:37.448+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:38.208 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:37.936+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:38.208 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:37.936+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:38.209 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:37.936+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:38.209 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:37.937+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:38.492 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:38.204+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:38.492 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:38.204+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:38.492 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:38.204+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:38.492 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:38.205+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:38.894 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:38.488+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:38.895 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:38.488+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:38.895 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:38.488+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:38.895 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:38.489+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:39.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:38.891+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:39.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:38.891+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:39.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:38.891+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:39.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:38.891+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:39.522 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:39.185+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:39.522 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:39.185+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:39.522 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:39.185+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:39.522 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:39.186+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:39.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:39.518+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:39.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:39.518+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:39.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:39.518+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:39.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:39.519+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:40.183 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:39.927+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:40.183 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:39.927+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:40.183 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:39.927+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:40.183 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:39.928+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:40.527 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:40.194+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:40.527 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:40.194+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:40.527 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:40.194+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:40.527 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:40.194+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:40.813 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:40.523+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:40.814 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:40.523+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:40.814 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:40.523+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:40.814 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:40.524+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:41.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:40.914+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:41.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:40.914+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:41.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:40.914+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:41.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:40.914+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:41.539 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:41.229+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:41.540 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:41.229+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:41.540 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:41.229+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:41.540 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:41.230+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:41.820 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:21:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:21:41.693+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 20251 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:21:41.821 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:41.536+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:41.821 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:41.536+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:41.821 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:41.536+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:41.821 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:41.537+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:42.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:41.937+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:42.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:41.937+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:42.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:41.937+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:42.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:41.938+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:42.504 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:42.214+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:42.504 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:42.214+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:42.505 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:42.214+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:42.505 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:42.215+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:42.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:42.500+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:42.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:42.500+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:42.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:42.500+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:42.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:42.501+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:43.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:42.968+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:43.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:42.968+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:43.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:42.968+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:43.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:42.969+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:43.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:43.185+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:43.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:43.185+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:43.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:43.185+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:43.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:43.186+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:43.886 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:43.514+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:43.887 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:43.514+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:43.887 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:43.514+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:43.887 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:43.514+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:44.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:44.004+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:44.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:44.004+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:44.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:44.004+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:44.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:44.005+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:44.566 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:44.207+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:44.566 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:44.207+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:44.566 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:44.207+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:44.567 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:44.208+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:44.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:44.562+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:44.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:44.562+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:44.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:44.562+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:44.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:44.563+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:45.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:44.992+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:45.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:44.992+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:45.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:44.992+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:45.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:44.993+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:45.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:45.220+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:45.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:45.220+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:45.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:45.220+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:45.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:45.221+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:45.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:45.550+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:45.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:45.550+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:45.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:45.550+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:45.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:45.551+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:46.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:46.215+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:46.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:46.215+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:46.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:46.215+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:46.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:46.216+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:46.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:46.030+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:46.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:46.030+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:46.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:46.030+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:46.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:46.031+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:46.826 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:21:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:21:46.695+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 20327 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:21:46.826 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:46.542+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:46.827 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:46.542+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:46.827 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:46.542+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:46.827 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:46.543+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:47.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:47.026+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:47.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:47.026+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:47.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:47.026+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:47.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:47.027+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:47.531 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:47.228+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:47.531 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:47.228+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:47.531 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:47.228+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:47.532 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:47.228+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:47.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:47.527+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:47.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:47.527+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:47.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:47.527+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:47.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:47.528+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:48.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:47.978+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:48.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:47.978+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:48.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:47.978+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:48.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:47.979+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:48.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:48.192+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:48.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:48.192+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:48.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:48.192+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:48.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:48.193+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:48.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:48.514+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:48.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:48.514+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:48.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:48.514+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:48.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:48.515+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:49.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:48.933+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:49.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:48.933+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:49.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:48.933+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:49.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:48.934+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:49.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:49.167+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:49.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:49.167+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:49.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:49.167+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:49.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:49.167+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:49.894 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:49.542+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:49.894 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:49.542+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:49.894 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:49.542+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:49.894 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:49.543+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:50.144 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:49.890+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:50.145 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:49.890+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:50.145 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:49.890+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:50.145 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:49.891+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:50.145 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:50.139+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:50.145 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:50.139+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:50.145 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:50.139+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:50.145 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:50.140+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:50.889 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:50.563+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:50.890 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:50.563+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:50.890 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:50.563+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:50.890 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:50.564+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:51.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:50.885+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:51.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:50.885+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:51.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:50.885+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:51.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:50.886+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:51.580 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:51.168+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:51.580 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:51.168+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:51.581 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:51.168+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:51.581 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:51.169+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:51.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:21:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:21:51.697+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 20409 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:21:51.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:51.576+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:51.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:51.576+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:51.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:51.576+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:51.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:51.577+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:52.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:51.916+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:52.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:51.916+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:52.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:51.916+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:52.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:51.917+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:52.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:52.154+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:52.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:52.154+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:52.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:52.154+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:52.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:52.155+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:52.900 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:52.554+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:52.900 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:52.554+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:52.900 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:52.554+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:52.900 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:52.555+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:53.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:52.896+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:53.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:52.896+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:53.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:52.896+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:53.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:52.897+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:53.498 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:53.182+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:53.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:53.182+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:53.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:53.182+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:53.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:53.183+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:53.871 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:53.581+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:53.872 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:53.581+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:53.872 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:53.581+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:53.872 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:53.582+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:54.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:53.867+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:54.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:53.867+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:54.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:53.867+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:54.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:53.869+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:54.636 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:54.202+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:54.636 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:54.202+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:54.636 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:54.202+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:54.636 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:54.203+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:54.888 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:54.632+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:54.888 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:54.632+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:54.888 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:54.632+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:54.888 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:54.633+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:55.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:54.884+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:55.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:54.884+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:55.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:54.884+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:55.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:54.885+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:55.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:55.180+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:55.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:55.180+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:55.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:55.180+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:55.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:55.181+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:56.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:55.886+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:56.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:55.886+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:56.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:55.886+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:56.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:55.887+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:56.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:55.662+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:56.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:55.662+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:56.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:55.662+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:56.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:55.663+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:56.634 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:56.186+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:56.634 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:56.186+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:56.634 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:56.186+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:56.634 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:56.187+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:56.890 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:21:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:21:56.699+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 20488 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:21:56.891 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:56.630+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:56.891 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:56.630+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:56.891 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:56.630+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:56.891 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:56.631+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:57.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:56.886+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:57.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:56.886+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:57.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:56.886+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:57.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:56.887+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:57.619 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:57.210+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:57.619 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:57.210+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:57.619 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:57.210+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:57.619 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:57.211+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:57.889 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:57.615+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:57.889 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:57.615+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:57.889 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:57.615+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:57.889 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:57.616+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:58.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:57.885+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:58.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:57.885+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:58.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:57.885+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:58.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:57.886+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:58.604 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:58.209+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:58.604 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:58.209+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:58.604 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:58.209+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:58.605 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:58.210+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:58.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:58.600+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:58.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:58.600+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:58.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:58.600+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:58.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:58.601+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:59.229 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:58.922+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:59.230 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:58.922+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:59.230 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:58.922+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:21:59.230 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:58.922+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:21:59.557 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:59.225+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:59.557 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:59.225+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:59.558 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:59.225+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:21:59.558 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:21:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:21:59.226+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:21:59.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:59.553+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:59.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:59.553+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:59.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:59.553+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:21:59.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:21:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:21:59.554+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:00.184 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:59.929+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:00.185 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:59.929+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:00.185 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:59.929+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:00.185 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:21:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:21:59.929+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:00.585 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:00.247+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:00.585 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:00.247+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:00.585 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:00.247+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:00.585 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:00.248+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:00.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:00.581+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:00.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:00.581+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:00.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:00.581+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:00.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:00.581+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:01.249 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:00.928+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:01.249 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:00.928+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:01.250 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:00.928+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:01.250 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:00.929+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:01.590 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:01.245+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:01.590 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:01.245+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:01.590 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:01.245+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:01.590 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:01.246+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:01.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:22:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:22:01.701+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 20573 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:22:01.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:01.586+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:01.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:01.586+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:01.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:01.586+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:01.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:01.587+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:02.236 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:01.917+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:02.236 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:01.917+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:02.236 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:01.917+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:02.237 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:01.918+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:02.560 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:02.232+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:02.560 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:02.232+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:02.560 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:02.232+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:02.561 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:02.233+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:02.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:02.556+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:02.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:02.556+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:02.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:02.556+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:02.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:02.557+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:03.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:02.955+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:03.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:02.955+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:03.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:02.955+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:03.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:02.956+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:03.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:03.196+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:03.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:03.196+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:03.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:03.196+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:03.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:03.197+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:03.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:03.535+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:03.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:03.535+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:03.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:03.535+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:03.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:03.535+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:04.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:03.949+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:04.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:03.949+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:04.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:03.949+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:04.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:03.951+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:04.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:04.148+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:04.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:04.148+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:04.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:04.148+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:04.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:04.149+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:04.906 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:04.534+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:04.906 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:04.534+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:04.906 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:04.534+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:04.906 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:04.535+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:05.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:04.902+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:05.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:04.902+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:05.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:04.902+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:05.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:04.903+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:05.498 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:05.161+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:05.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:05.161+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:05.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:05.161+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:05.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:05.162+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:05.877 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:05.494+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:05.877 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:05.495+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:05.877 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:05.495+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:05.877 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:05.495+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:06.142 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:05.873+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:06.142 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:05.873+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:06.142 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:05.873+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:06.142 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:05.873+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:06.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:06.138+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:06.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:06.138+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:06.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:06.138+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:06.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:06.139+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:06.720 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:22:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:22:06.703+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 20655 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:22:06.721 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:06.454+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:06.721 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:06.454+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:06.721 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:06.454+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:06.721 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:06.455+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:07.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:06.843+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:07.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:06.843+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:07.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:06.843+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:07.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:06.844+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:07.452 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:07.168+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:07.452 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:07.168+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:07.452 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:07.168+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:07.452 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:07.169+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:07.865 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:07.448+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:07.865 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:07.448+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:07.865 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:07.448+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:07.865 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:07.449+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:08.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:07.861+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:08.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:07.861+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:08.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:07.861+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:08.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:07.861+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:08.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:08.426+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:08.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:08.426+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:08.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:08.426+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:08.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:08.427+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:08.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:08.204+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:08.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:08.204+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:08.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:08.204+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:08.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:08.205+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:09.227 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:08.909+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:09.228 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:08.909+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:09.228 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:08.909+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:09.228 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:08.910+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:09.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:09.457+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:09.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:09.457+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:09.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:09.457+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:09.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:09.458+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:09.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:09.224+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:09.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:09.224+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:09.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:09.224+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:09.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:09.225+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:10.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:09.899+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:10.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:09.899+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:10.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:09.899+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:10.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:09.900+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:10.482 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:10.186+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:10.482 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:10.186+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:10.483 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:10.186+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:10.483 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:10.187+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:10.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:10.478+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:10.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:10.478+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:10.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:10.478+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:10.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:10.479+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:11.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:10.940+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:11.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:10.940+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:11.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:10.940+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:11.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:10.941+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:11.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:11.182+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:11.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:11.182+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:11.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:11.182+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:11.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:11.183+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:11.833 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:22:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:22:11.705+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 20739 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:22:11.834 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:11.507+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:11.834 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:11.507+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:11.834 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:11.507+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:11.834 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:11.508+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:12.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:11.990+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:12.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:11.990+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:12.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:11.990+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:12.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:11.990+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:12.506 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:12.173+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:12.506 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:12.173+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:12.506 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:12.173+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:12.506 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:12.173+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:12.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:12.502+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:12.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:12.502+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:12.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:12.502+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:12.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:12.503+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:13.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:13.038+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:13.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:13.038+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:13.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:13.038+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:13.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:13.039+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:13.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:13.138+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:13.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:13.138+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:13.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:13.138+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:13.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:13.139+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:13.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:13.550+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:13.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:13.550+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:13.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:13.550+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:13.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:13.551+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:14.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:14.160+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:14.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:14.160+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:14.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:14.160+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:14.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:14.161+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:14.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:14.029+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:14.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:14.029+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:14.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:14.029+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:14.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:14.030+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:14.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:14.597+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:14.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:14.597+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:14.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:14.597+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:14.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:14.598+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:15.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:15.057+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:15.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:15.057+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:15.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:15.057+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:15.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:15.057+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:15.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:15.177+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:15.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:15.177+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:15.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:15.177+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:15.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:15.178+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:15.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:15.639+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:15.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:15.639+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:15.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:15.639+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:15.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:15.640+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:16.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:16.065+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:16.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:16.065+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:16.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:16.065+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:16.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:16.066+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:16.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:16.186+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:16.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:16.186+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:16.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:16.186+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:16.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:16.187+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:16.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:22:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:22:16.707+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 20821 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:22:16.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:16.649+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:16.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:16.649+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:16.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:16.649+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:16.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:16.650+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:17.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:17.056+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:17.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:17.056+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:17.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:17.056+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:17.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:17.057+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:17.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:17.194+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:17.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:17.194+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:17.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:17.194+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:17.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:17.195+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:17.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:17.617+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:17.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:17.617+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:17.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:17.617+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:17.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:17.617+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:18.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:18.039+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:18.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:18.039+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:18.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:18.039+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:18.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:18.040+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:18.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:18.237+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:18.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:18.237+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:18.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:18.237+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:18.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:18.238+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:18.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:18.629+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:18.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:18.629+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:18.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:18.629+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:18.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:18.630+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:19.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:19.085+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:19.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:19.085+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:19.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:19.085+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:19.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:19.086+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:19.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:19.206+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:19.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:19.206+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:19.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:19.206+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:19.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:19.206+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:19.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:19.614+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:19.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:19.614+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:19.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:19.614+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:19.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:19.615+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:20.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:20.163+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:20.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:20.163+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:20.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:20.163+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:20.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:20.164+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:20.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:20.036+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:20.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:20.036+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:20.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:20.036+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:20.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:20.037+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:20.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:20.614+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:20.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:20.614+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:20.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:20.614+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:20.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:20.615+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:21.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:21.144+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:21.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:21.144+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:21.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:21.144+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:21.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:21.145+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:21.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:21.038+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:21.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:21.038+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:21.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:21.038+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:21.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:21.039+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:21.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:22:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:22:21.708+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 20906 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:22:21.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:21.634+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:21.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:21.634+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:21.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:21.634+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:21.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:21.635+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:22.165+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:22.165+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:22.165+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:22.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:22.166+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:22.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:21.991+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:22.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:21.991+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:22.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:21.991+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:22.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:21.991+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:22.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:22.586+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:22.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:22.586+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:22.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:22.586+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:22.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:22.587+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:23.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:23.028+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:23.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:23.028+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:23.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:23.028+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:23.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:23.029+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:23.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:23.176+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:23.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:23.176+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:23.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:23.176+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:23.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:23.177+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:23.822 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:23.547+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:23.822 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:23.547+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:23.822 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:23.547+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:23.822 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:23.547+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:24.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:23.987+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:24.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:23.987+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:24.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:23.987+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:24.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:23.988+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:24.572 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:24.224+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:24.573 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:24.224+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:24.573 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:24.224+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:24.573 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:24.225+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:24.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:24.568+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:24.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:24.568+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:24.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:24.568+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:24.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:24.568+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:25.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:25.015+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:25.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:25.015+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:25.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:25.015+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:25.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:25.016+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:25.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:25.251+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:25.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:25.251+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:25.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:25.251+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:25.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:25.252+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:25.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:25.604+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:25.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:25.604+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:25.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:25.604+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:25.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:25.604+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:26.288 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:26.024+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:26.288 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:26.024+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:26.288 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:26.024+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:26.289 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:26.025+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:26.641 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:26.284+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:26.642 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:26.284+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:26.642 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:26.284+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:26.642 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:26.284+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:26.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:22:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:22:26.712+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 20988 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:22:26.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:26.638+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:26.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:26.638+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:26.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:26.638+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:26.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:26.638+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:27.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:27.248+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:27.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:27.248+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:27.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:27.248+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:27.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:27.249+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:27.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:27.016+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:27.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:27.016+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:27.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:27.016+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:27.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:27.017+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:28.025 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:27.678+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:28.025 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:27.678+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:28.025 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:27.678+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:28.025 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:27.679+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:28.293 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:28.021+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:28.294 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:28.021+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:28.294 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:28.021+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:28.294 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:28.022+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:28.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:28.290+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:28.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:28.290+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:28.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:28.290+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:28.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:28.291+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:28.989 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:28.707+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:28.989 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:28.707+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:28.989 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:28.707+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:28.990 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:28.708+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:29.279 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:28.985+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:29.279 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:28.985+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:29.279 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:28.985+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:29.279 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:28.986+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:29.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:29.275+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:29.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:29.275+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:29.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:29.275+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:29.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:29.276+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:29.968 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:29.658+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:29.969 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:29.658+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:29.969 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:29.658+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:29.969 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:29.659+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:30.281 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:29.965+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:30.282 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:29.965+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:30.282 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:29.965+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:30.282 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:29.966+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:30.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:30.278+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:30.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:30.278+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:30.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:30.278+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:30.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:30.278+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:30.945 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:30.662+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:30.945 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:30.662+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:30.945 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:30.662+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:30.946 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:30.662+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:31.302 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:30.941+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:31.302 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:30.941+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:31.303 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:30.941+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:31.303 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:30.942+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:31.626 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:31.298+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:31.627 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:31.298+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:31.627 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:31.298+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:31.627 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:31.300+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:31.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:22:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:22:31.714+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 21072 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:22:31.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:31.623+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:31.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:31.623+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:31.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:31.623+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:31.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:31.624+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:32.313 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:31.914+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:32.313 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:31.914+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:32.314 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:31.914+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:32.314 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:31.915+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:32.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:32.309+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:32.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:32.309+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:32.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:32.309+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:32.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:32.311+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:33.038 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:32.873+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:33.038 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:32.873+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:33.038 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:32.873+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:33.038 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:32.874+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:33.038 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:32.659+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:33.039 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:32.659+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:33.039 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:32.659+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:33.039 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:32.660+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:33.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:33.280+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:33.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:33.280+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:33.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:33.280+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:33.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:33.281+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:33.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:33.654+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:33.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:33.654+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:33.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:33.654+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:33.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:33.654+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:34.288 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:33.920+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:34.289 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:33.920+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:34.289 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:33.920+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:34.289 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:33.921+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:34.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:34.285+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:34.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:34.285+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:34.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:34.285+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:34.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:34.286+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:34.936 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:34.678+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:34.937 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:34.678+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:34.937 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:34.678+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:34.937 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:34.678+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:35.274 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:34.932+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:35.275 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:34.932+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:35.275 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:34.932+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:35.275 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:34.933+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:35.654 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:35.271+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:35.655 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:35.271+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:35.655 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:35.271+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:35.655 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:35.271+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:35.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:35.889+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:35.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:35.889+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:35.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:35.889+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:35.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:35.889+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:35.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:35.651+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:35.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:35.651+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:35.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:35.651+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:35.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:35.652+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:36.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:36.258+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:36.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:36.258+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:36.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:36.258+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:36.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:36.258+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:36.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:22:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:22:36.716+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 21153 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:22:36.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:36.633+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:36.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:36.633+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:36.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:36.633+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:36.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:36.634+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:37.226 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:36.925+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:37.226 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:36.925+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:37.226 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:36.925+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:37.226 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:36.926+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:37.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:37.222+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:37.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:37.222+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:37.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:37.222+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:37.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:37.223+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:37.945 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:37.662+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:37.945 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:37.662+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:37.945 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:37.662+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:37.945 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:37.663+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:38.242 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:37.941+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:38.242 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:37.941+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:38.242 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:37.941+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:38.243 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:37.942+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:38.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:38.238+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:38.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:38.238+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:38.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:38.238+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:38.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:38.239+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:39.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:38.700+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:39.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:38.700+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:39.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:38.700+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:39.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:38.701+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:39.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:38.901+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:39.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:38.901+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:39.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:38.901+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:39.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:38.901+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:39.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:39.259+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:39.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:39.259+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:39.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:39.259+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:39.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:39.259+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:40.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:39.889+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:40.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:39.889+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:40.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:39.889+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:40.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:39.890+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:40.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:39.691+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:40.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:39.691+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:40.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:39.691+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:40.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:39.692+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:40.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:40.239+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:40.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:40.239+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:40.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:40.239+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:40.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:40.240+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:41.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:40.723+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:41.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:40.723+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:41.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:40.723+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:41.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:40.724+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:41.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:40.920+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:41.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:40.920+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:41.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:40.920+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:41.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:40.921+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:41.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:41.264+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:41.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:41.264+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:41.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:41.264+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:41.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:41.265+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:41.964 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:22:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:22:41.718+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 21238 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:22:41.965 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:41.703+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:41.965 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:41.703+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:41.965 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:41.703+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:41.965 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:41.704+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:42.254 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:41.961+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:42.254 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:41.961+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:42.254 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:41.961+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:42.254 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:41.961+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:42.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:42.263+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:42.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:42.263+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:42.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:42.263+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:42.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:42.263+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:43.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:42.749+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:43.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:42.749+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:43.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:42.749+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:43.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:42.750+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:43.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:42.978+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:43.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:42.978+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:43.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:42.978+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:43.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:42.979+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:43.498 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:43.228+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:43.498 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:43.228+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:43.498 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:43.228+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:43.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:43.229+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:44.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:44.004+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:44.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:44.004+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:44.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:44.004+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:44.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:44.004+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:44.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:43.772+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:44.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:43.772+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:44.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:43.772+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:44.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:43.773+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:44.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:44.183+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:44.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:44.183+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:44.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:44.183+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:44.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:44.185+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:45.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:45.013+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:45.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:45.013+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:45.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:45.013+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:45.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:45.014+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:45.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:44.812+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:45.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:44.812+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:45.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:44.812+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:45.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:44.814+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:45.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:45.227+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:45.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:45.227+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:45.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:45.227+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:45.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:45.228+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:46.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:45.981+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:46.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:45.981+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:46.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:45.981+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:46.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:45.981+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:46.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:45.788+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:46.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:45.788+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:46.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:45.788+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:46.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:45.789+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:46.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:46.235+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:46.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:46.235+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:46.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:46.235+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:46.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:46.235+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:46.997 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:22:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:22:46.720+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 21320 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:22:46.998 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:46.832+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:46.998 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:46.832+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:46.998 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:46.832+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:46.998 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:46.832+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:47.248 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:46.993+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:47.248 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:46.993+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:47.248 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:46.993+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:47.249 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:46.994+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:47.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:47.244+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:47.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:47.244+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:47.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:47.244+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:47.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:47.245+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:48.030 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:47.811+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:48.030 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:47.811+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:48.031 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:47.811+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:48.031 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:47.812+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:48.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:48.026+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:48.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:48.026+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:48.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:48.026+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:48.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:48.027+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:48.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:48.258+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:48.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:48.258+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:48.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:48.258+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:48.410 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:48.259+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:49.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:48.828+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:49.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:48.828+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:49.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:48.828+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:49.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:48.829+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:49.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:49.049+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:49.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:49.049+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:49.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:49.049+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:49.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:49.050+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:49.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:49.251+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:49.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:49.251+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:49.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:49.251+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:49.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:49.252+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:50.063 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:49.785+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:50.064 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:49.785+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:50.064 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:49.785+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:50.064 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:49.786+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:50.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:50.060+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:50.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:50.060+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:50.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:50.060+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:50.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:50.061+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:50.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:50.282+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:50.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:50.282+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:50.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:50.282+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:50.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:50.283+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:51.096 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:50.817+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:51.096 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:50.817+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:51.097 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:50.817+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:51.097 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:50.818+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:51.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:51.092+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:51.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:51.092+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:51.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:51.092+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:51.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:51.093+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:51.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:51.248+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:51.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:51.248+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:51.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:51.248+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:51.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:51.249+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:52.142 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:22:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:22:51.722+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 21405 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:22:52.143 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:51.820+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:52.143 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:51.820+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:52.143 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:51.820+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:52.143 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:51.821+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:52.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:52.138+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:52.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:52.138+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:52.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:52.138+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:52.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:52.140+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:52.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:52.243+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:52.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:52.243+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:52.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:52.243+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:52.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:52.244+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:53.142 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:52.863+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:53.142 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:52.863+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:53.142 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:52.863+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:53.142 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:52.863+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:53.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:53.232+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:53.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:53.232+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:53.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:53.232+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:53.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:53.233+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:53.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:53.138+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:53.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:53.138+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:53.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:53.138+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:53.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:53.139+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:54.172 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:53.908+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:54.172 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:53.908+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:54.172 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:53.908+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:54.172 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:53.908+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:54.542 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:54.168+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:54.542 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:54.168+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:54.542 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:54.168+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:54.542 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:54.169+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:54.543 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:54.217+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:54.543 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:54.217+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:54.543 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:54.217+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:54.543 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:54.218+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:55.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:54.894+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:55.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:54.894+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:55.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:54.894+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:55.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:54.895+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:55.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:55.217+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:55.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:55.217+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:55.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:55.217+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:55.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:55.218+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:55.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:55.235+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:55.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:55.235+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:55.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:55.235+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:55.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:55.236+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:56.205 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:55.921+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:56.205 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:55.921+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:56.205 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:55.921+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:56.206 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:55.921+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:56.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:56.201+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:56.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:56.201+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:56.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:56.201+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:56.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:56.202+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:56.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:56.216+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:56.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:56.216+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:56.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:56.216+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:56.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:56.217+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:57.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:22:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:22:56.724+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 21487 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:22:57.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:56.945+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:57.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:56.945+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:57.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:56.945+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:57.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:56.946+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:57.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:57.239+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:57.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:57.239+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:57.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:57.239+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:57.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:57.240+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:57.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:57.154+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:57.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:57.154+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:57.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:57.154+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:57.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:57.155+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:58.179 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:57.917+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:58.180 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:57.917+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:58.180 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:57.917+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:58.180 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:57.918+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:58.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:58.239+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:58.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:58.239+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:58.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:58.239+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:58.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:58.240+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:58.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:58.176+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:58.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:58.176+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:58.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:58.176+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:58.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:58.177+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:59.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:58.900+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:59.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:58.900+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:59.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:58.900+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:22:59.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:58.901+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:22:59.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:59.130+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:59.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:59.130+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:59.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:59.130+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:22:59.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:22:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:22:59.131+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:22:59.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:59.191+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:59.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:59.191+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:59.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:59.191+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:22:59.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:22:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:22:59.192+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:00.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:00.112+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:00.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:00.112+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:00.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:00.112+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:00.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:00.113+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:00.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:59.894+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:00.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:59.894+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:00.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:59.894+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:00.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:22:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:22:59.894+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:00.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:00.203+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:00.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:00.203+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:00.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:00.203+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:00.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:00.204+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:01.113 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:00.896+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:01.113 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:00.896+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:01.113 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:00.896+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:01.114 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:00.897+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:01.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:01.220+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:01.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:01.220+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:01.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:01.220+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:01.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:01.221+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:01.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:01.109+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:01.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:01.109+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:01.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:01.109+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:01.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:01.110+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:02.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:23:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:23:01.727+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 21571 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:23:02.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:01.923+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:02.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:01.923+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:02.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:01.923+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:02.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:01.924+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:02.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:02.156+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:02.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:02.156+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:02.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:02.156+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:02.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:02.157+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:02.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:02.240+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:02.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:02.240+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:02.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:02.240+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:02.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:02.240+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:03.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:02.897+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:03.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:02.897+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:03.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:02.897+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:03.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:02.898+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:03.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:03.242+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:03.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:03.242+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:03.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:03.242+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:03.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:03.243+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:03.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:03.189+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:03.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:03.189+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:03.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:03.189+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:03.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:03.190+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:04.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:03.936+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:04.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:03.936+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:04.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:03.936+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:04.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:03.937+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:04.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:04.264+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:04.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:04.264+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:04.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:04.264+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:04.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:04.264+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:04.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:04.158+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:04.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:04.158+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:04.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:04.158+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:04.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:04.158+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:05.278 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:04.939+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:05.278 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:04.939+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:05.278 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:04.939+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:05.278 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:04.940+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:05.278 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:05.131+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:05.279 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:05.131+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:05.279 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:05.131+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:05.279 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:05.132+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:05.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:05.274+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:05.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:05.274+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:05.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:05.274+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:05.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:05.275+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:06.253 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:05.933+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:06.254 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:05.933+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:06.254 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:05.933+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:06.254 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:05.933+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:06.254 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:06.177+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:06.254 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:06.177+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:06.254 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:06.177+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:06.255 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:06.177+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:06.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:06.250+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:06.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:06.250+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:06.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:06.250+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:06.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:06.251+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:07.146 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:23:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:23:06.729+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 21653 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:23:07.146 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:06.927+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:07.147 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:06.927+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:07.147 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:06.927+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:07.147 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:06.927+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:07.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:07.240+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:07.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:07.240+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:07.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:07.240+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:07.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:07.241+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:07.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:07.142+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:07.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:07.142+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:07.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:07.142+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:07.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:07.143+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:08.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:07.894+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:08.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:07.894+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:08.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:07.894+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:08.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:07.895+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:08.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:08.122+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:08.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:08.122+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:08.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:08.122+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:08.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:08.123+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:08.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:08.195+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:08.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:08.195+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:08.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:08.195+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:08.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:08.196+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:09.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:08.852+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:09.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:08.852+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:09.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:08.852+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:09.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:08.853+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:09.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:09.198+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:09.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:09.198+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:09.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:09.198+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:09.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:09.199+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:09.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:09.168+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:09.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:09.168+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:09.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:09.168+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:09.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:09.169+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:10.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:09.874+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:10.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:09.874+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:10.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:09.874+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:10.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:09.875+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:10.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:10.230+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:10.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:10.230+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:10.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:10.230+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:10.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:10.230+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:10.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:10.205+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:10.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:10.205+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:10.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:10.205+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:10.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:10.205+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:11.187 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:10.919+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:11.188 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:10.919+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:11.188 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:10.919+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:11.188 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:10.920+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:11.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:11.184+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:11.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:11.184+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:11.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:11.184+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:11.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:11.185+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:11.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:11.192+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:11.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:11.192+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:11.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:11.192+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:11.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:11.193+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:12.147 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:23:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:23:11.731+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 21737 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:23:12.148 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:11.893+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:12.148 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:11.893+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:12.148 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:11.893+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:12.148 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:11.893+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:12.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:12.162+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:12.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:12.162+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:12.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:12.162+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:12.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:12.162+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:12.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:12.144+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:12.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:12.144+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:12.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:12.144+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:12.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:12.144+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:13.212 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:13.133+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:13.212 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:13.133+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:13.212 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:13.133+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:13.212 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:13.134+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:13.213 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:12.939+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:13.213 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:12.939+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:13.213 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:12.939+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:13.213 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:12.939+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:13.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:13.208+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:13.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:13.208+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:13.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:13.208+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:13.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:13.209+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:14.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:14.162+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:14.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:14.162+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:14.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:14.162+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:14.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:14.163+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:14.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:14.129+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:14.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:14.129+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:14.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:14.129+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:14.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:14.130+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:14.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:13.981+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:14.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:13.981+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:14.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:13.981+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:14.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:13.982+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:15.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:15.185+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:15.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:15.185+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:15.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:15.185+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:15.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:15.186+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:15.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:15.104+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:15.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:15.104+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:15.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:15.104+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:15.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:15.104+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:15.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:14.986+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:15.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:14.986+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:15.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:14.986+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:15.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:14.987+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:16.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:16.026+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:16.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:16.026+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:16.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:16.026+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:16.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:16.027+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:16.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:16.155+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:16.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:16.155+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:16.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:16.155+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:16.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:16.156+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:16.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:16.086+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:16.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:16.086+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:16.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:16.086+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:16.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:16.087+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:17.060 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:23:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:23:16.733+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 21819 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:23:17.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:17.104+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:17.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:17.104+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:17.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:17.104+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:17.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:17.104+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:17.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:17.056+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:17.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:17.056+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:17.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:17.056+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:17.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:17.057+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:17.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:17.141+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:17.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:17.141+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:17.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:17.141+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:17.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:17.142+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:18.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:18.136+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:18.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:18.136+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:18.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:18.136+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:18.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:18.137+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:18.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:18.059+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:18.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:18.059+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:18.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:18.059+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:18.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:18.060+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:18.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:18.127+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:18.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:18.127+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:18.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:18.127+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:18.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:18.128+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:19.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:19.157+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:19.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:19.157+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:19.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:19.157+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:19.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:19.158+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:19.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:19.066+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:19.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:19.066+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:19.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:19.066+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:19.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:19.067+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:19.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:19.137+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:19.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:19.137+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:19.410 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:19.137+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:19.410 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:19.137+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:20.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:20.165+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:20.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:20.165+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:20.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:20.165+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:20.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:20.165+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:20.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:20.085+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:20.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:20.085+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:20.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:20.085+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:20.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:20.086+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:20.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:20.094+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:20.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:20.094+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:20.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:20.094+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:20.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:20.095+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:21.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:21.114+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:21.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:21.114+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:21.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:21.114+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:21.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:21.115+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:21.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:21.211+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:21.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:21.212+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:21.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:21.212+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:21.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:21.213+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:21.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:21.053+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:21.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:21.053+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:21.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:21.053+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:21.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:21.053+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:22.047 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:23:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:23:21.735+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 21904 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:23:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:22.043+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:22.043+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:22.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:22.043+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:22.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:22.044+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:22.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:22.083+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:22.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:22.083+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:22.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:22.083+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:22.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:22.083+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:22.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:22.188+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:22.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:22.188+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:22.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:22.188+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:22.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:22.189+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:23.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:23.214+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:23.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:23.214+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:23.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:23.214+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:23.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:23.215+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:23.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:23.036+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:23.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:23.036+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:23.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:23.036+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:23.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:23.037+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:23.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:23.091+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:23.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:23.091+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:23.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:23.091+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:23.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:23.092+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:24.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:24.103+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:24.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:24.103+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:24.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:24.103+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:24.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:24.103+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:24.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:24.039+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:24.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:24.039+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:24.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:24.039+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:24.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:24.040+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:24.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:24.221+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:24.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:24.221+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:24.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:24.221+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:24.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:24.221+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:25.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:25.149+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:25.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:25.149+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:25.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:25.149+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:25.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:25.149+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:25.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:25.197+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:25.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:25.197+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:25.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:25.197+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:25.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:25.198+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:25.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:25.089+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:25.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:25.089+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:25.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:25.089+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:25.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:25.091+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:26.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:26.203+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:26.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:26.203+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:26.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:26.203+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:26.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:26.204+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:26.110+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:26.110+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:26.110+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:26.111+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:26.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:26.195+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:26.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:26.195+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:26.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:26.195+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:26.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:26.197+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:27.080 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:23:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:23:26.738+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 21986 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:23:27.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:27.240+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:27.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:27.240+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:27.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:27.240+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:27.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:27.241+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:27.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:27.076+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:27.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:27.076+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:27.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:27.076+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:27.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:27.077+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:27.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:27.194+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:27.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:27.194+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:27.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:27.194+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:27.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:27.195+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:28.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:28.198+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:28.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:28.198+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:28.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:28.198+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:28.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:28.199+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:28.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:28.194+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:28.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:28.194+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:28.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:28.194+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:28.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:28.195+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:28.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:28.089+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:28.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:28.089+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:28.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:28.089+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:28.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:28.090+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:29.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:29.233+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:29.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:29.233+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:29.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:29.233+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:29.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:29.233+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:29.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:29.127+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:29.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:29.127+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:29.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:29.127+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:29.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:29.128+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:29.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:29.197+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:29.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:29.197+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:29.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:29.197+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:29.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:29.198+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:30.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:30.207+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:30.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:30.207+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:30.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:30.207+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:30.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:30.208+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:30.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:30.097+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:30.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:30.097+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:30.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:30.097+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:30.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:30.097+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:30.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:30.240+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:30.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:30.240+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:30.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:30.240+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:30.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:30.241+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:31.406 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:31.184+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:31.406 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:31.184+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:31.406 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:31.184+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:31.406 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:31.185+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:31.406 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:31.289+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:31.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:31.289+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:31.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:31.289+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:31.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:31.291+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:31.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:31.096+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:31.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:31.096+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:31.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:31.096+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:31.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:31.097+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:32.119 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:23:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:23:31.740+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 22071 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:23:32.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:32.204+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:32.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:32.204+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:32.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:32.204+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:32.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:32.205+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:32.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:32.115+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:32.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:32.115+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:32.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:32.115+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:32.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:32.116+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:32.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:32.283+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:32.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:32.283+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:32.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:32.283+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:32.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:32.284+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:33.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:33.158+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:33.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:33.158+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:33.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:33.158+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:33.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:33.159+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:33.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:33.139+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:33.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:33.139+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:33.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:33.139+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:33.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:33.139+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:33.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:33.255+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:33.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:33.255+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:33.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:33.255+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:33.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:33.256+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:34.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:34.150+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:34.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:34.150+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:34.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:34.150+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:34.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:34.150+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:34.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:34.266+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:34.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:34.266+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:34.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:34.266+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:34.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:34.267+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:34.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:34.141+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:34.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:34.141+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:34.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:34.141+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:34.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:34.142+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:35.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:35.189+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:35.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:35.189+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:35.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:35.189+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:35.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:35.190+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:35.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:35.219+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:35.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:35.219+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:35.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:35.219+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:35.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:35.220+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:35.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:35.181+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:35.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:35.181+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:35.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:35.181+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:35.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:35.182+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:36.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:36.147+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:36.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:36.147+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:36.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:36.147+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:36.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:36.148+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:36.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:36.254+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:36.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:36.254+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:36.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:36.254+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:36.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:36.255+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:36.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:36.184+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:36.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:36.184+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:36.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:36.184+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:36.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:36.185+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:37.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:23:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:23:36.742+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 22153 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:23:37.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:37.180+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:37.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:37.180+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:37.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:37.180+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:37.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:37.181+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:37.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:37.298+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:37.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:37.298+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:37.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:37.298+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:37.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:37.299+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:37.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:37.184+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:37.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:37.184+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:37.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:37.184+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:37.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:37.185+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:38.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:38.183+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:38.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:38.183+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:38.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:38.183+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:38.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:38.185+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:38.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:38.346+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:38.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:38.346+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:38.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:38.346+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:38.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:38.347+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:38.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:38.177+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:38.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:38.177+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:38.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:38.177+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:38.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:38.178+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:39.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:39.141+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:39.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:39.141+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:39.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:39.141+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:39.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:39.142+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:39.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:39.309+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:39.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:39.309+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:39.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:39.309+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:39.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:39.310+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:39.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:39.158+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:39.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:39.158+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:39.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:39.158+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:39.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:39.160+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:40.315 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:40.101+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:40.315 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:40.101+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:40.315 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:40.101+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:40.316 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:40.103+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:40.316 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:40.149+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:40.316 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:40.149+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:40.316 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:40.149+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:40.316 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:40.150+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:40.622 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:40.311+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:40.622 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:40.311+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:40.622 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:40.311+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:40.622 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:40.312+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:41.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:41.143+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:41.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:41.143+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:41.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:41.143+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:41.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:41.144+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:41.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:41.361+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:41.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:41.361+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:41.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:41.361+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:41.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:41.362+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:41.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:41.142+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:41.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:41.142+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:41.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:41.142+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:41.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:41.142+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:42.128 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:23:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:23:41.745+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 22238 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:23:42.389 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:42.125+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:42.390 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:42.125+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:42.390 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:42.125+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:42.390 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:42.125+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:42.390 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:42.172+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:42.390 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:42.172+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:42.391 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:42.172+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:42.391 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:42.173+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:42.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:42.386+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:42.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:42.386+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:42.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:42.386+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:42.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:42.387+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:43.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:43.094+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:43.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:43.094+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:43.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:43.094+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:43.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:43.094+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:43.219+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:43.219+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:43.219+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:43.220+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:43.694 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:43.419+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:43.694 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:43.419+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:43.694 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:43.419+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:43.694 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:43.420+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:44.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:44.045+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:44.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:44.045+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:44.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:44.045+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:44.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:44.046+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:44.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:44.188+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:44.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:44.188+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:44.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:44.188+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:44.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:44.188+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:44.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:44.442+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:44.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:44.442+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:44.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:44.442+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:44.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:44.442+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:45.401 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:45.054+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:45.402 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:45.054+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:45.402 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:45.054+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:45.402 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:45.055+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:45.402 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:45.186+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:45.402 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:45.186+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:45.403 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:45.186+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:45.403 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:45.187+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:45.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:45.398+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:45.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:45.398+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:45.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:45.398+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:45.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:45.398+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:46.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:46.010+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:46.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:46.010+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:46.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:46.010+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:46.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:46.011+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:46.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:46.174+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:46.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:46.174+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:46.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:46.174+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:46.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:46.174+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:46.751 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:46.424+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:46.751 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:46.424+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:46.752 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:46.424+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:46.752 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:46.425+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:47.064 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:23:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:23:46.747+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 22320 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:23:47.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:47.061+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:47.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:47.061+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:47.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:47.061+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:47.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:47.061+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:47.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:47.201+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:47.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:47.201+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:47.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:47.201+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:47.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:47.202+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:47.849 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:47.420+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:47.849 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:47.420+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:47.850 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:47.420+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:47.850 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:47.421+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:48.373 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:48.103+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:48.373 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:48.103+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:48.373 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:48.103+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:48.373 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:48.104+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:48.373 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:48.195+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:48.374 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:48.195+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:48.374 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:48.195+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:48.374 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:48.196+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:48.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:48.407+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:48.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:48.407+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:48.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:48.407+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:48.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:48.408+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:49.376 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:49.079+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:49.376 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:49.079+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:49.376 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:49.079+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:49.376 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:49.080+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:49.377 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:49.167+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:49.377 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:49.167+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:49.377 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:49.167+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:49.377 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:49.168+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:49.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:49.449+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:49.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:49.449+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:49.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:49.449+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:49.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:49.450+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:50.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:50.088+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:50.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:50.088+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:50.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:50.088+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:50.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:50.088+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:50.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:50.158+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:50.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:50.158+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:50.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:50.158+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:50.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:50.159+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:50.766 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:50.463+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:50.766 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:50.463+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:50.766 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:50.463+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:50.767 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:50.464+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:51.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:51.127+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:51.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:51.127+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:51.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:51.127+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:51.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:51.128+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:51.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:51.109+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:51.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:51.109+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:51.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:51.109+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:51.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:51.110+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:51.728 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:51.465+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:51.728 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:51.465+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:51.729 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:51.465+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:51.729 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:51.465+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:52.085 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:23:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:23:51.749+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 22405 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:23:52.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:52.084+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:52.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:52.084+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:52.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:52.084+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:52.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:52.086+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:52.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:52.080+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:52.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:52.080+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:52.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:52.080+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:52.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:52.081+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:52.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:52.419+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:52.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:52.419+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:52.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:52.419+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:52.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:52.420+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:53.376 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:53.132+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:53.377 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:53.132+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:53.377 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:53.132+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:53.377 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:53.133+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:53.378 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:53.042+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:53.378 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:53.042+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:53.378 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:53.042+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:53.378 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:53.043+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:53.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:53.372+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:53.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:53.372+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:53.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:53.372+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:53.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:53.373+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:54.372 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:54.139+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:54.372 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:54.139+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:54.372 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:54.139+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:54.372 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:54.140+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:54.372 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:54.083+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:54.373 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:54.083+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:54.373 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:54.083+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:54.373 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:54.084+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:54.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:54.368+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:54.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:54.368+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:54.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:54.368+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:54.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:54.369+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:55.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:55.135+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:55.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:55.135+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:55.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:55.135+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:55.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:55.136+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:55.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:55.039+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:55.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:55.039+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:55.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:55.039+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:55.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:55.040+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:55.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:55.404+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:55.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:55.404+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:55.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:55.404+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:55.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:55.405+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:56.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:56.091+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:56.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:56.091+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:56.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:56.091+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:56.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:56.091+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:56.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:56.061+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:56.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:56.061+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:56.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:56.061+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:56.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:56.062+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:56.755 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:56.434+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:56.756 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:56.434+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:56.756 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:56.434+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:56.756 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:56.435+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:57.061 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:23:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:23:56.750+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 22487 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:23:57.404 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:57.108+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:57.405 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:57.108+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:57.405 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:57.108+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:57.405 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:57.108+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:57.405 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:57.056+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:57.405 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:57.057+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:57.405 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:57.057+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:57.405 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:57.057+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:57.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:57.399+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:57.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:57.399+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:57.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:57.399+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:57.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:57.400+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:58.359 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:58.087+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:58.360 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:58.087+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:58.360 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:58.087+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:58.360 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:58.088+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:58.360 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:58.012+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:58.360 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:58.012+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:58.361 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:58.012+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:58.361 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:58.013+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:58.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:58.354+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:58.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:58.354+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:58.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:58.354+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:58.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:58.355+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:23:59.190 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:59.095+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:59.190 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:59.095+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:59.190 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:59.095+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:23:59.190 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:23:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:23:59.096+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:59.191 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:59.008+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:59.191 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:59.008+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:59.191 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:59.008+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:23:59.191 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:23:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:23:59.009+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:23:59.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:59.391+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:59.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:59.391+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:59.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:59.391+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:23:59.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:23:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:23:59.392+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:00.386 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:00.123+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:00.386 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:00.123+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:00.386 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:00.123+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:00.386 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:00.124+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:00.387 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:00.054+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:00.387 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:00.054+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:00.387 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:00.054+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:00.387 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:00.055+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:00.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:00.381+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:00.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:00.381+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:00.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:00.381+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:00.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:00.382+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:01.404 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:01.163+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:01.404 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:01.163+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:01.404 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:01.163+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:01.405 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:01.163+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:01.405 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:01.016+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:01.405 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:01.016+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:01.406 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:01.016+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:01.406 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:01.017+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:01.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:01.398+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:01.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:01.398+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:01.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:01.398+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:01.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:01.399+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:02.017 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:24:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:24:01.753+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 22572 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:24:02.369 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:02.189+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:02.369 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:02.189+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:02.370 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:02.189+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:02.370 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:02.191+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:02.370 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:02.012+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:02.370 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:02.012+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:02.370 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:02.012+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:02.371 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:02.014+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:02.371 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:02.364+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:02.371 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:02.364+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:02.371 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:02.364+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:02.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:02.366+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:03.386 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:03.195+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:03.386 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:03.195+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:03.386 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:03.195+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:03.386 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:03.196+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:03.387 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:03.002+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:03.387 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:03.002+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:03.387 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:03.002+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:03.387 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:03.003+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:03.387 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:03.381+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:03.388 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:03.381+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:03.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:03.381+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:03.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:03.382+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:04.369 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:04.207+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:04.369 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:04.207+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:04.370 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:04.207+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:04.370 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:04.208+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:04.370 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:04.013+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:04.370 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:04.013+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:04.370 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:04.013+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:04.370 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:04.014+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:04.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:04.365+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:04.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:04.365+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:04.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:04.365+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:04.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:04.366+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:05.321 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:05.182+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:05.321 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:05.182+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:05.321 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:05.182+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:05.322 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:05.183+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:05.322 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:04.989+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:05.322 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:04.989+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:05.322 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:04.989+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:05.322 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:04.990+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:05.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:05.317+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:05.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:05.317+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:05.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:05.317+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:05.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:05.318+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:06.283 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:06.218+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:06.284 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:06.218+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:06.284 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:06.218+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:06.284 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:06.219+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:06.284 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:05.990+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:06.284 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:05.990+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:06.285 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:05.990+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:06.285 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:05.990+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:06.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:06.280+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:06.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:06.280+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:06.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:06.280+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:06.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:06.281+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:07.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:24:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:24:06.756+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 22654 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:24:07.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:06.991+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:07.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:06.991+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:07.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:06.991+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:07.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:06.992+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:07.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:07.245+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:07.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:07.245+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:07.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:07.245+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:07.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:07.246+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:07.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:07.297+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:07.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:07.297+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:07.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:07.297+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:07.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:07.299+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:08.273 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:07.971+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:08.274 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:07.971+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:08.274 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:07.971+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:08.274 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:07.972+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:08.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:08.269+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:08.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:08.269+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:08.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:08.269+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:08.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:08.270+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:08.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:08.301+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:08.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:08.301+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:08.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:08.301+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:08.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:08.301+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:09.288 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:08.974+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:09.288 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:08.974+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:09.288 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:08.974+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:09.289 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:08.975+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:09.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:09.284+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:09.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:09.284+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:09.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:09.284+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:09.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:09.284+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:09.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:09.305+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:09.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:09.305+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:09.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:09.305+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:09.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:09.306+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:10.317 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:09.956+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:10.317 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:09.956+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:10.317 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:09.956+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:10.317 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:09.957+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:10.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:10.313+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:10.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:10.313+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:10.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:10.313+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:10.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:10.314+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:10.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:10.322+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:10.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:10.322+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:10.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:10.322+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:10.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:10.323+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:11.283 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:10.937+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:11.283 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:10.937+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:11.284 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:10.937+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:11.284 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:10.938+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:11.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:11.279+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:11.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:11.279+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:11.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:11.279+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:11.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:11.280+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:11.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:11.304+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:11.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:11.304+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:11.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:11.304+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:11.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:11.304+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:12.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:24:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:24:11.758+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 22739 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:24:12.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:11.902+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:12.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:11.902+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:12.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:11.902+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:12.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:11.902+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:12.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:12.259+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:12.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:12.259+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:12.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:12.259+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:12.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:12.260+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:12.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:12.353+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:12.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:12.353+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:12.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:12.353+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:12.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:12.354+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:13.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:12.882+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:13.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:12.882+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:13.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:12.882+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:13.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:12.882+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:13.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:13.213+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:13.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:13.213+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:13.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:13.213+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:13.500 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:13.214+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:13.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:13.306+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:13.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:13.306+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:13.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:13.306+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:13.501 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:13.307+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:14.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:13.877+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:14.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:13.877+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:14.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:13.877+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:14.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:13.877+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:14.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:14.227+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:14.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:14.227+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:14.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:14.227+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:14.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:14.228+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:14.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:14.290+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:14.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:14.290+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:14.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:14.290+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:14.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:14.291+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:15.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:14.861+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:15.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:14.861+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:15.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:14.862+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:15.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:14.862+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:15.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:15.269+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:15.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:15.269+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:15.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:15.269+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:15.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:15.270+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:15.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:15.302+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:15.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:15.302+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:15.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:15.302+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:15.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:15.303+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:16.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:15.898+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:16.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:15.898+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:16.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:15.898+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:16.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:15.899+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:16.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:16.308+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:16.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:16.308+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:16.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:16.308+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:16.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:16.309+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:16.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:16.300+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:16.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:16.300+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:16.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:16.300+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:16.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:16.301+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:17.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:24:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:24:16.761+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 22821 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:24:17.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:16.883+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:17.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:16.883+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:17.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:16.883+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:17.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:16.884+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:17.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:17.290+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:17.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:17.290+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:17.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:17.290+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:17.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:17.290+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:17.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:17.262+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:17.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:17.262+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:17.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:17.262+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:17.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:17.263+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:18.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:17.884+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:18.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:17.884+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:18.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:17.884+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:18.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:17.885+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:18.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:18.217+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:18.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:18.217+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:18.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:18.217+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:18.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:18.218+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:18.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:18.324+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:18.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:18.324+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:18.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:18.324+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:18.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:18.325+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:19.203 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:18.913+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:19.203 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:18.913+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:19.203 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:18.913+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:19.204 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:18.914+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:19.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:19.199+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:19.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:19.199+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:19.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:19.199+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:19.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:19.200+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:19.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:19.326+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:19.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:19.326+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:19.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:19.326+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:19.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:19.327+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:20.193 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:19.904+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:20.193 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:19.904+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:20.194 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:19.904+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:20.194 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:19.905+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:20.558 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:20.250+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:20.558 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:20.250+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:20.558 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:20.250+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:20.558 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:20.250+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:20.558 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:20.368+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:20.558 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:20.368+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:20.559 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:20.368+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:20.559 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:20.370+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:21.260 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:20.932+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:21.260 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:20.932+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:21.260 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:20.932+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:21.261 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:20.932+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:21.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:21.256+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:21.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:21.256+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:21.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:21.256+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:21.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:21.257+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:21.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:21.395+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:21.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:21.395+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:21.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:21.395+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:21.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:21.396+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:22.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:24:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:24:21.763+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 22906 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:24:22.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:21.934+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:22.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:21.934+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:22.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:21.934+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:22.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:21.935+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:22.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:22.236+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:22.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:22.236+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:22.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:22.236+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:22.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:22.238+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:22.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:22.411+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:22.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:22.411+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:22.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:22.411+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:22.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:22.412+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:23.217 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:22.917+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:23.218 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:22.917+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:23.218 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:22.917+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:23.218 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:22.918+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:23.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:23.213+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:23.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:23.213+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:23.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:23.213+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:23.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:23.214+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:23.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:23.431+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:23.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:23.431+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:23.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:23.431+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:23.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:23.432+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:24.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:24.175+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:24.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:24.175+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:24.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:24.175+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:24.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:24.176+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:24.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:23.964+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:24.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:23.964+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:24.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:23.964+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:24.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:23.964+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:24.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:24.440+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:24.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:24.440+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:24.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:24.440+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:24.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:24.441+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:25.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:25.187+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:25.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:25.187+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:25.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:25.187+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:25.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:25.188+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:25.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:24.980+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:25.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:24.980+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:25.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:24.980+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:25.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:24.980+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:25.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:25.447+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:25.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:25.447+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:25.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:25.447+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:25.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:25.448+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:26.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:26.205+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:26.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:26.205+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:26.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:26.205+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:26.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:26.206+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:26.008+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:26.008+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:26.008+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:26.009+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:26.769 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:26.448+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:26.769 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:26.448+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:26.769 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:26.448+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:26.770 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:26.448+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:27.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:24:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:24:26.765+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 22988 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:24:27.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:26.980+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:27.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:26.980+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:27.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:26.980+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:27.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:26.981+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:27.472 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:27.193+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:27.472 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:27.193+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:27.472 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:27.193+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:27.472 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:27.194+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:27.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:27.468+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:27.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:27.468+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:27.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:27.468+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:27.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:27.469+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:28.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:28.151+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:28.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:28.151+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:28.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:28.151+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:28.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:28.152+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:28.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:27.988+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:28.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:27.988+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:28.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:27.988+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:28.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:27.989+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:28.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:28.420+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:28.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:28.420+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:28.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:28.420+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:28.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:28.421+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:29.398 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:29.133+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:29.399 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:29.133+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:29.399 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:29.133+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:29.399 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:29.134+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:29.399 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:28.975+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:29.399 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:28.975+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:29.399 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:28.975+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:29.400 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:28.976+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:29.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:29.394+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:29.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:29.394+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:29.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:29.394+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:29.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:29.395+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:30.405 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:30.164+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:30.405 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:30.164+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:30.405 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:30.164+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:30.405 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:30.165+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:30.405 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:29.959+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:30.406 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:29.959+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:30.406 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:29.959+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:30.406 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:29.960+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:30.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:30.401+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:30.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:30.401+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:30.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:30.401+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:30.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:30.402+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:31.214 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:30.936+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:31.214 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:30.936+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:31.214 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:30.936+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:31.214 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:30.937+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:31.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:31.210+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:31.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:31.210+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:31.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:31.210+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:31.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:31.211+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:31.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:31.389+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:31.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:31.389+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:31.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:31.389+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:31.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:31.390+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:32.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:24:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:24:31.767+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 23073 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:24:32.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:31.896+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:32.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:31.896+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:32.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:31.896+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:32.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:31.896+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:32.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:32.223+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:32.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:32.223+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:32.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:32.223+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:32.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:32.224+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:32.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:32.405+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:32.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:32.405+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:32.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:32.405+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:32.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:32.405+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:33.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:32.894+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:33.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:32.894+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:33.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:32.894+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:33.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:32.895+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:33.498 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:33.177+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:33.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:33.177+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:33.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:33.177+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:33.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:33.177+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:33.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:33.401+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:33.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:33.401+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:33.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:33.401+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:33.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:33.402+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:34.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:33.855+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:34.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:33.855+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:34.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:33.855+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:34.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:33.855+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:34.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:34.356+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:34.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:34.356+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:34.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:34.356+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:34.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:34.357+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:34.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:34.223+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:34.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:34.223+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:34.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:34.223+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:34.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:34.223+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:35.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:34.898+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:35.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:34.898+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:35.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:34.898+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:35.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:34.899+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:35.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:35.327+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:35.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:35.327+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:35.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:35.327+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:35.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:35.328+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:35.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:35.235+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:35.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:35.235+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:35.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:35.235+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:35.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:35.236+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:36.198 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:35.934+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:36.199 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:35.934+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:36.199 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:35.934+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:36.199 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:35.935+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:36.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:36.195+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:36.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:36.195+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:36.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:36.195+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:36.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:36.195+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:36.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:36.360+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:36.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:36.360+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:36.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:36.360+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:36.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:36.361+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:37.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:24:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:24:36.770+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 23155 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:24:37.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:36.919+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:37.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:36.919+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:37.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:36.919+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:37.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:36.920+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:37.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:37.393+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:37.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:37.393+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:37.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:37.393+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:37.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:37.394+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:37.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:37.172+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:37.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:37.172+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:37.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:37.172+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:37.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:37.173+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:38.207 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:37.950+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:38.207 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:37.950+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:38.207 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:37.950+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:38.207 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:37.951+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:38.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:38.203+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:38.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:38.203+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:38.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:38.203+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:38.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:38.204+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:38.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:38.421+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:38.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:38.421+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:38.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:38.421+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:38.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:38.422+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:39.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:39.165+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:39.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:39.165+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:39.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:39.165+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:39.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:39.165+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:39.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:38.940+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:39.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:38.940+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:39.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:38.940+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:39.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:38.941+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:39.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:39.440+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:39.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:39.440+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:39.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:39.440+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:39.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:39.441+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:40.396 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:40.117+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:40.396 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:40.117+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:40.396 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:40.117+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:40.396 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:40.118+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:40.397 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:39.948+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:40.397 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:39.948+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:40.397 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:39.948+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:40.397 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:39.949+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:40.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:40.392+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:40.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:40.392+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:40.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:40.392+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:40.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:40.393+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:41.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:41.124+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:41.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:41.124+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:41.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:41.124+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:41.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:41.125+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:41.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:40.973+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:41.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:40.973+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:41.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:40.973+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:41.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:40.974+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:41.731 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:41.409+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:41.731 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:41.409+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:41.731 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:41.409+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:41.731 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:41.409+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:42.062 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:24:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:24:41.772+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 23262 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:24:42.062 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:41.959+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:42.062 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:41.959+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:42.062 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:41.959+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:42.063 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:41.960+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:42.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:42.095+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:42.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:42.095+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:42.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:42.095+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:42.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:42.096+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:42.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:42.429+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:42.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:42.429+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:42.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:42.429+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:42.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:42.430+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:43.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:43.122+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:43.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:43.122+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:43.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:43.122+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:43.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:43.123+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:42.954+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:42.954+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:42.954+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:42.955+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:43.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:43.456+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:43.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:43.456+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:43.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:43.456+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:43.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:43.456+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:44.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:44.131+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:44.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:44.131+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:44.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:44.131+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:44.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:44.132+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:44.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:43.955+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:44.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:43.955+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:44.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:43.955+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:44.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:43.956+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:44.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:44.484+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:44.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:44.484+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:44.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:44.484+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:44.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:44.485+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:45.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:45.176+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:45.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:45.176+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:45.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:45.176+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:45.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:45.176+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:45.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:44.995+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:45.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:44.995+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:45.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:44.995+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:45.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:44.996+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:45.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:45.489+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:45.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:45.489+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:45.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:45.489+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:45.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:45.490+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:46.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:46.147+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:46.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:46.147+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:46.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:46.147+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:46.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:46.148+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:46.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:45.967+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:46.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:45.967+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:46.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:45.967+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:46.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:45.967+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:46.802 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:24:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:24:46.775+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 23344 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:24:46.803 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:46.539+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:46.803 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:46.539+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:46.803 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:46.539+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:46.803 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:46.540+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:47.118 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:46.998+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:47.119 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:46.998+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:47.119 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:46.998+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:47.119 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:46.999+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:47.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:47.115+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:47.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:47.115+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:47.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:47.115+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:47.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:47.116+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:47.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:47.556+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:47.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:47.556+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:47.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:47.556+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:47.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:47.557+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:48.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:48.160+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:48.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:48.160+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:48.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:48.160+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:48.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:48.160+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:48.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:47.987+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:48.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:47.987+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:48.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:47.987+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:48.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:47.988+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:48.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:48.519+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:48.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:48.519+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:48.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:48.519+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:48.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:48.520+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:49.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:48.967+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:49.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:48.967+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:49.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:48.967+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:49.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:48.968+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:49.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:49.148+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:49.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:49.148+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:49.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:49.148+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:49.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:49.149+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:49.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:49.507+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:49.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:49.507+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:49.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:49.507+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:49.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:49.507+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:50.195 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:50.157+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:50.195 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:50.157+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:50.195 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:50.157+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:50.196 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:50.158+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:50.196 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:49.923+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:50.196 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:49.923+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:50.196 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:49.923+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:50.196 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:49.924+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:50.890 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:50.552+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:50.890 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:50.552+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:50.890 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:50.552+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:50.890 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:50.553+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:51.150 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:50.886+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:51.150 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:50.886+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:51.150 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:50.886+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:51.150 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:50.887+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:51.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:51.146+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:51.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:51.146+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:51.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:51.146+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:51.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:51.147+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:51.864 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:24:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:24:51.777+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 23428 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:24:51.865 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:51.586+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:51.865 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:51.586+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:51.865 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:51.586+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:51.865 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:51.587+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:52.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:51.861+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:52.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:51.861+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:52.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:51.861+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:52.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:51.862+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:52.621 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:52.185+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:52.621 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:52.185+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:52.621 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:52.185+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:52.621 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:52.186+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:52.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:52.841+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:52.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:52.841+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:52.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:52.841+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:52.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:52.842+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:52.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:52.617+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:52.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:52.617+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:52.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:52.617+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:52.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:52.618+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:53.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:53.166+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:53.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:53.166+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:53.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:53.166+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:53.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:53.167+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:53.852 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:53.655+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:53.852 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:53.655+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:53.852 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:53.655+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:53.852 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:53.656+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:54.148 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:53.848+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:54.148 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:53.848+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:54.148 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:53.848+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:54.149 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:53.849+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:54.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:54.144+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:54.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:54.144+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:54.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:54.144+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:54.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:54.145+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:54.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:54.616+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:54.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:54.616+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:54.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:54.616+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:54.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:54.616+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:54.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:54.853+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:54.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:54.853+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:54.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:54.853+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:54.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:54.854+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:55.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:55.141+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:55.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:55.141+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:55.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:55.141+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:55.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:55.141+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:55.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:55.834+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:55.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:55.834+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:55.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:55.834+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:55.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:55.835+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:55.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:55.600+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:55.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:55.600+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:55.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:55.600+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:55.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:55.601+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:56.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:56.134+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:56.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:56.134+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:56.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:56.134+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:56.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:56.135+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:56.866 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:24:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:24:56.780+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 23511 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:24:56.867 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:56.596+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:56.867 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:56.596+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:56.867 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:56.596+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:56.867 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:56.596+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:57.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:56.862+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:57.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:56.862+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:57.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:56.862+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:57.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:56.863+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:57.619 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:57.172+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:57.619 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:57.172+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:57.619 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:57.172+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:57.619 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:57.173+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:57.891 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:57.615+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:57.892 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:57.615+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:57.892 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:57.615+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:57.892 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:57.615+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:58.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:57.887+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:58.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:57.887+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:58.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:57.887+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:58.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:57.888+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:58.634 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:58.191+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:58.634 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:58.191+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:58.635 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:58.191+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:58.635 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:58.192+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:58.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:58.630+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:58.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:58.630+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:58.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:58.630+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:58.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:58.631+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:24:59.195 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:58.930+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:59.195 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:58.930+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:59.195 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:58.930+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:24:59.196 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:58.931+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:59.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:59.191+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:59.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:59.191+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:59.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:59.191+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:24:59.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:24:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:24:59.192+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:24:59.935 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:59.672+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:59.935 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:59.672+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:59.935 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:59.672+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:24:59.936 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:24:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:24:59.672+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:00.195 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:00.176+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:00.196 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:00.176+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:00.196 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:00.176+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:00.196 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:00.177+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:00.196 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:59.931+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:00.196 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:59.931+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:00.196 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:59.931+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:00.197 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:24:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:24:59.932+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:00.896 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:00.680+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:00.896 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:00.680+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:00.897 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:00.680+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:00.897 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:00.681+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:01.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:00.892+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:01.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:00.892+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:01.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:00.892+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:01.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:00.893+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:01.651 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:01.224+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:01.651 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:01.224+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:01.651 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:01.224+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:01.651 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:01.225+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:01.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:25:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:25:01.782+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 23595 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:25:01.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:01.647+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:01.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:01.647+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:01.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:01.647+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:01.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:01.648+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:02.254 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:01.911+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:02.254 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:01.911+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:02.254 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:01.911+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:02.255 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:01.912+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:02.641 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:02.250+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:02.641 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:02.250+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:02.641 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:02.250+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:02.641 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:02.251+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:02.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:02.637+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:02.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:02.637+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:02.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:02.637+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:02.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:02.638+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:03.230 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:02.906+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:03.231 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:02.906+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:03.231 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:02.906+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:03.231 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:02.906+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:03.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:03.227+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:03.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:03.227+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:03.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:03.227+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:03.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:03.227+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:03.887 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:03.668+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:03.887 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:03.668+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:03.888 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:03.668+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:03.888 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:03.670+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:04.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:03.883+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:04.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:03.883+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:04.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:03.883+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:04.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:03.884+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:04.647 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:04.235+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:04.647 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:04.235+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:04.647 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:04.235+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:04.647 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:04.236+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:04.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:04.643+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:04.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:04.643+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:04.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:04.643+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:04.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:04.644+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:04.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:04.844+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:04.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:04.844+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:04.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:04.844+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:04.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:04.845+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:05.606 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:05.235+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:05.606 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:05.235+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:05.606 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:05.235+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:05.607 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:05.235+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:05.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:05.602+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:05.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:05.602+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:05.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:05.602+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:05.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:05.603+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:05.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:05.806+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:05.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:05.806+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:05.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:05.806+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:05.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:05.807+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:06.639 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:06.232+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:06.639 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:06.232+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:06.639 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:06.232+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:06.639 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:06.233+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:06.907 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:25:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:25:06.784+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 23678 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:25:06.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:06.635+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:06.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:06.635+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:06.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:06.635+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:06.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:06.636+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:06.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:06.841+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:06.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:06.841+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:06.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:06.841+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:06.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:06.842+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:07.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:07.214+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:07.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:07.214+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:07.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:07.214+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:07.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:07.215+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:08.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:07.826+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:08.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:07.826+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:08.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:07.826+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:08.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:07.826+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:08.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:07.658+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:08.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:07.658+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:08.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:07.658+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:08.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:07.659+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:08.639 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:08.253+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:08.639 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:08.253+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:08.639 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:08.253+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:08.640 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:08.253+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:08.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:08.635+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:08.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:08.635+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:08.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:08.635+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:08.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:08.636+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:08.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:08.824+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:08.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:08.824+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:08.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:08.824+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:08.909 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:08.825+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:09.641 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:09.298+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:09.641 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:09.298+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:09.642 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:09.298+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:09.642 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:09.299+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:09.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:09.846+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:09.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:09.846+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:09.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:09.846+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:09.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:09.847+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:09.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:09.637+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:09.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:09.637+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:09.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:09.637+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:09.910 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:09.638+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:10.642 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:10.340+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:10.642 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:10.340+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:10.642 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:10.340+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:10.642 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:10.341+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:10.896 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:10.638+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:10.896 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:10.638+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:10.897 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:10.638+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:10.897 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:10.639+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:11.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:10.892+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:11.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:10.892+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:11.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:10.892+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:11.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:10.893+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:11.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:11.293+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:11.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:11.293+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:11.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:11.293+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:11.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:11.293+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:11.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:25:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:25:11.787+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 23762 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:25:11.920 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:11.655+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:11.921 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:11.655+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:11.921 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:11.655+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:11.921 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:11.656+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:12.299 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:11.929+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:12.300 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:11.929+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:12.300 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:11.929+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:12.300 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:11.930+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:12.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:12.295+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:12.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:12.296+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:12.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:12.296+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:12.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:12.296+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:13.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:12.889+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:13.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:12.889+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:13.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:12.889+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:13.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:12.890+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:13.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:12.676+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:13.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:12.676+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:13.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:12.676+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:13.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:12.677+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:13.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:13.316+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:13.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:13.316+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:13.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:13.316+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:13.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:13.317+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:14.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:13.888+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:14.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:13.888+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:14.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:13.888+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:14.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:13.889+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:14.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:13.700+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:14.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:13.700+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:14.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:13.700+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:14.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:13.701+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:14.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:14.362+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:14.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:14.362+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:14.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:14.362+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:14.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:14.363+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:15.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:14.870+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:15.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:14.870+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:15.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:14.870+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:15.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:14.871+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:15.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:14.659+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:15.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:14.659+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:15.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:14.659+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:15.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:14.660+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:15.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:15.361+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:15.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:15.361+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:15.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:15.361+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:15.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:15.362+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:16.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:15.914+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:16.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:15.914+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:16.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:15.914+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:16.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:15.915+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:16.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:15.682+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:16.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:15.682+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:16.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:15.682+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:16.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:15.683+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:16.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:16.328+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:16.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:16.328+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:16.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:16.328+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:16.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:16.329+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:16.920 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:25:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:25:16.789+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 23845 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:25:16.921 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:16.664+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:16.921 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:16.664+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:16.921 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:16.664+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:16.921 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:16.665+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:16.921 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:16.909+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:16.922 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:16.909+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:16.922 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:16.909+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:16.922 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:16.910+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:17.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:17.358+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:17.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:17.358+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:17.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:17.358+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:17.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:17.359+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:17.951 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:17.688+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:17.951 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:17.688+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:17.951 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:17.688+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:17.951 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:17.689+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:18.388 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:17.947+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:18.389 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:17.947+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:18.389 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:17.947+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:18.389 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:17.948+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:18.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:18.384+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:18.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:18.384+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:18.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:18.384+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:18.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:18.385+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:19.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:18.982+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:19.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:18.982+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:19.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:18.982+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:19.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:18.983+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:19.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:18.733+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:19.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:18.733+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:19.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:18.733+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:19.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:18.734+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:19.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:19.342+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:19.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:19.342+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:19.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:19.342+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:19.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:19.343+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:20.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:19.941+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:20.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:19.941+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:20.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:19.941+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:20.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:19.942+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:20.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:19.761+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:20.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:19.761+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:20.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:19.761+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:20.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:19.762+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:20.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:20.374+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:20.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:20.374+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:20.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:20.374+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:20.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:20.375+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:21.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:20.923+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:21.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:20.923+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:21.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:20.923+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:21.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:20.923+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:21.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:20.729+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:21.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:20.729+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:21.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:20.729+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:21.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:20.730+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:21.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:21.367+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:21.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:21.367+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:21.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:21.367+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:21.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:21.367+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:21.998 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:25:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:25:21.792+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 23929 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:25:21.998 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:21.777+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:21.998 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:21.777+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:21.998 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:21.777+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:21.998 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:21.778+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:21.999 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:21.921+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:21.999 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:21.921+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:21.999 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:21.921+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:21.999 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:21.921+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:22.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:22.356+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:22.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:22.356+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:22.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:22.356+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:22.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:22.357+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:23.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:22.959+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:23.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:22.959+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:23.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:22.959+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:23.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:22.959+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:23.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:22.827+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:23.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:22.827+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:23.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:22.827+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:23.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:22.828+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:23.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:23.362+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:23.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:23.362+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:23.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:23.362+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:23.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:23.364+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:24.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:23.862+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:24.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:23.862+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:24.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:23.862+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:24.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:23.862+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:24.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:23.992+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:24.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:23.992+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:24.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:23.992+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:24.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:23.993+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:24.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:24.392+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:24.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:24.392+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:24.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:24.392+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:24.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:24.392+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:25.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:25.017+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:25.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:25.017+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:25.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:25.017+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:25.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:25.018+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:25.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:24.819+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:25.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:24.819+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:25.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:24.819+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:25.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:24.820+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:25.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:25.380+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:25.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:25.380+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:25.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:25.380+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:25.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:25.380+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:26.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:26.056+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:26.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:26.056+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:26.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:26.056+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:26.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:26.057+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:26.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:25.838+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:26.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:25.838+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:26.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:25.838+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:26.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:25.839+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:26.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:26.354+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:26.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:26.354+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:26.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:26.354+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:26.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:26.354+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:27.052 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:25:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:25:26.794+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 24012 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:25:27.053 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:27.049+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:27.053 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:27.049+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:27.053 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:26.859+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:27.053 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:26.859+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:27.053 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:26.859+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:27.054 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:26.860+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:27.401 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:27.049+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:27.401 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:27.049+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:27.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:27.397+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:27.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:27.397+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:27.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:27.397+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:27.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:27.398+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:28.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:28.024+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:28.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:28.024+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:28.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:28.024+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:28.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:28.025+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:28.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:27.821+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:28.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:27.821+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:28.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:27.821+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:28.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:27.822+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:28.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:28.360+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:28.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:28.360+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:28.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:28.360+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:28.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:28.361+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:29.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:28.826+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:29.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:28.826+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:29.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:28.826+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:29.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:28.827+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:29.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:29.016+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:29.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:29.016+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:29.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:29.016+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:29.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:29.017+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:29.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:29.319+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:29.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:29.319+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:29.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:29.319+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:29.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:29.320+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:30.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:29.875+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:30.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:29.875+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:30.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:29.875+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:30.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:29.876+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:30.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:29.999+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:30.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:29.999+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:30.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:29.999+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:30.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:30.000+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:30.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:30.345+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:30.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:30.345+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:30.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:30.345+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:30.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:30.346+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:31.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:30.850+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:31.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:30.850+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:31.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:30.850+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:31.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:30.851+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:31.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:30.990+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:31.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:30.990+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:31.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:30.990+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:31.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:30.991+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:31.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:31.337+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:31.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:31.337+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:31.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:31.337+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:31.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:31.338+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:31.999 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:25:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:25:31.797+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 24096 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:25:32.000 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:31.850+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:32.000 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:31.850+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:32.000 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:31.850+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:32.000 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:31.851+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:32.359 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:31.995+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:32.360 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:31.995+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:32.360 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:31.995+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:32.360 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:31.996+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:32.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:32.355+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:32.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:32.355+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:32.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:32.355+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:32.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:32.356+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:33.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:32.977+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:33.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:32.977+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:33.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:32.977+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:33.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:32.978+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:33.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:32.844+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:33.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:32.844+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:33.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:32.844+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:33.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:32.845+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:33.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:33.340+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:33.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:33.340+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:33.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:33.340+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:33.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:33.341+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:34.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:33.982+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:34.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:33.982+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:34.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:33.982+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:34.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:33.983+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:34.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:33.888+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:34.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:33.888+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:34.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:33.888+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:34.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:33.889+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:34.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:34.377+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:34.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:34.377+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:34.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:34.377+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:34.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:34.377+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:35.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:34.962+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:35.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:34.963+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:35.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:34.963+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:35.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:34.964+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:35.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:34.869+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:35.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:34.869+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:35.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:34.869+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:35.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:34.869+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:35.899 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:35.413+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:35.900 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:35.413+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:35.900 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:35.413+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:35.900 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:35.413+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:36.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:35.895+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:36.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:35.895+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:36.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:35.895+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:36.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:35.896+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:36.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:35.930+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:36.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:35.930+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:36.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:35.930+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:36.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:35.930+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:36.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:36.400+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:36.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:36.400+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:36.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:36.400+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:36.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:36.400+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:37.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:25:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:25:36.798+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 24179 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:25:37.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:36.940+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:37.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:36.940+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:37.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:36.940+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:37.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:36.940+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:37.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:36.935+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:37.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:36.935+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:37.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:36.935+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:37.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:36.936+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:37.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:37.389+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:37.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:37.389+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:37.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:37.389+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:37.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:37.390+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:38.381 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:37.939+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:38.381 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:37.939+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:38.381 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:37.939+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:38.381 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:37.940+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:38.382 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:37.946+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:38.382 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:37.946+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:38.382 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:37.946+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:38.382 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:37.946+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:38.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:38.376+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:38.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:38.376+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:38.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:38.376+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:38.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:38.377+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:39.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:38.978+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:39.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:38.978+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:39.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:38.978+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:39.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:38.978+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:39.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:38.943+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:39.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:38.943+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:39.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:38.943+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:39.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:38.944+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:39.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:39.425+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:39.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:39.425+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:39.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:39.425+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:39.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:39.426+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:40.198 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:39.982+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:40.198 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:39.982+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:40.199 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:39.982+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:40.199 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:39.983+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:40.199 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:39.941+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:40.199 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:39.941+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:40.199 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:39.941+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:40.199 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:39.942+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:40.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:40.456+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:40.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:40.456+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:40.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:40.456+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:40.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:40.457+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:41.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:41.029+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:41.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:41.029+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:41.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:41.029+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:41.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:41.029+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:41.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:40.966+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:41.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:40.966+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:41.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:40.966+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:41.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:40.967+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:41.734 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:41.469+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:41.734 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:41.469+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:41.734 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:41.469+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:41.734 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:41.469+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:42.048 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:25:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:25:41.801+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 24263 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:25:42.049 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:41.920+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:42.049 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:41.920+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:42.049 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:41.920+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:42.049 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:41.921+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:42.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:42.045+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:42.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:42.045+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:42.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:42.045+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:42.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:42.046+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:42.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:42.489+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:42.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:42.489+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:42.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:42.489+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:42.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:42.490+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:43.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:42.946+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:43.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:42.946+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:43.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:42.946+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:43.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:42.946+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:43.057+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:43.057+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:43.057+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:43.058+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:43.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:43.504+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:43.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:43.504+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:43.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:43.504+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:43.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:43.505+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:44.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:43.934+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:44.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:43.934+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:44.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:43.934+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:44.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:43.935+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:44.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:44.047+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:44.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:44.047+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:44.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:44.047+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:44.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:44.048+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:44.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:44.544+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:44.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:44.544+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:44.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:44.544+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:44.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:44.545+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:45.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:45.086+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:45.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:45.086+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:45.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:45.086+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:45.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:45.086+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:45.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:44.942+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:45.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:44.943+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:45.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:44.943+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:45.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:44.943+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:45.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:45.503+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:45.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:45.503+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:45.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:45.503+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:45.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:45.504+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:46.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:45.934+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:46.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:45.934+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:46.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:45.934+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:46.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:45.935+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:46.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:46.046+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:46.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:46.046+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:46.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:46.046+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:46.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:46.047+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:46.808 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:46.474+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:46.808 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:46.474+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:46.808 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:46.474+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:46.809 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:46.474+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:47.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:25:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:25:46.804+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 24346 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:25:47.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:47.011+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:47.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:47.011+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:47.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:47.011+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:47.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:47.012+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:47.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:46.966+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:47.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:46.966+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:47.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:46.966+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:47.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:46.966+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:47.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:47.450+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:47.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:47.451+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:47.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:47.451+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:47.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:47.452+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:48.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:48.017+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:48.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:48.017+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:48.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:48.017+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:48.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:48.018+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:48.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:48.007+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:48.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:48.007+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:48.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:48.007+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:48.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:48.008+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:48.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:48.412+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:48.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:48.412+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:48.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:48.412+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:48.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:48.413+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:49.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:49.006+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:49.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:49.006+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:49.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:49.006+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:49.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:49.007+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:49.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:49.014+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:49.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:49.014+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:49.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:49.014+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:49.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:49.015+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:49.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:49.403+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:49.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:49.403+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:49.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:49.403+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:49.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:49.404+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:50.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:49.992+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:50.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:49.992+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:50.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:49.992+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:50.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:49.993+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:50.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:50.015+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:50.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:50.015+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:50.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:50.015+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:50.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:50.016+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:50.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:50.408+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:50.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:50.408+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:50.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:50.408+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:50.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:50.409+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:51.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:51.063+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:51.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:51.063+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:51.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:51.063+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:51.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:51.064+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:51.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:50.990+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:51.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:50.990+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:51.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:50.990+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:51.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:50.991+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:51.734 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:51.450+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:51.734 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:51.450+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:51.734 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:51.450+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:51.735 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:51.452+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:52.013 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:25:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:25:51.807+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 24430 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:25:52.406 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:52.053+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:52.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:52.053+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:52.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:52.053+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:52.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:52.054+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:52.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:52.009+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:52.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:52.009+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:52.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:52.009+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:52.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:52.010+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:52.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:52.403+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:52.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:52.403+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:52.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:52.403+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:52.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:52.403+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:53.371 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:53.042+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:53.371 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:53.042+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:53.372 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:53.042+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:53.372 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:53.044+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:53.372 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:52.995+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:53.372 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:52.995+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:53.372 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:52.995+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:53.372 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:52.996+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:53.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:53.367+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:53.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:53.367+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:53.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:53.367+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:53.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:53.368+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:54.389 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:54.055+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:54.390 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:54.055+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:54.390 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:54.055+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:54.390 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:54.056+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:54.390 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:53.999+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:54.390 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:53.999+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:54.390 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:53.999+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:54.391 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:53.999+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:54.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:54.386+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:54.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:54.386+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:54.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:54.386+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:54.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:54.387+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:55.399 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:54.958+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:55.399 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:54.958+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:55.400 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:54.958+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:55.400 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:54.958+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:55.400 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:55.090+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:55.401 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:55.090+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:55.401 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:55.090+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:55.401 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:55.090+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:55.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:55.395+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:55.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:55.395+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:55.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:55.395+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:55.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:55.396+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:56.380 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:56.094+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:56.380 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:56.094+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:56.380 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:56.094+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:56.381 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:56.094+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:56.381 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:55.912+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:56.381 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:55.912+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:56.381 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:55.912+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:56.381 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:55.913+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:56.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:56.376+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:56.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:56.376+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:56.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:56.376+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:56.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:56.377+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:57.106 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:25:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:25:56.809+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 24513 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:25:57.106 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:56.885+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:57.106 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:56.885+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:57.106 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:56.885+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:57.106 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:56.886+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:57.379 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:57.102+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:57.380 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:57.102+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:57.380 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:57.102+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:57.380 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:57.103+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:57.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:57.376+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:57.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:57.376+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:57.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:57.376+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:57.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:57.377+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:58.102 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:57.844+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:58.102 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:57.844+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:58.102 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:57.844+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:58.102 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:57.845+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:58.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:58.341+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:58.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:58.341+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:58.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:58.341+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:58.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:58.342+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:58.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:58.098+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:58.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:58.098+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:58.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:58.098+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:58.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:58.099+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:59.103 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:58.844+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:59.104 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:58.844+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:59.104 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:58.844+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:25:59.104 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:58.845+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:25:59.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:59.304+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:59.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:59.304+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:59.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:59.304+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:25:59.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:25:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:25:59.305+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:25:59.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:59.100+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:59.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:59.100+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:59.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:59.100+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:25:59.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:25:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:25:59.100+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:00.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:00.099+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:00.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:00.099+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:00.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:00.099+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:00.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:00.100+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:00.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:59.877+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:00.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:59.877+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:00.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:59.877+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:00.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:25:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:25:59.878+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:00.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:00.321+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:00.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:00.321+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:00.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:00.321+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:00.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:00.322+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:01.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:01.065+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:01.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:01.065+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:01.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:01.065+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:01.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:01.065+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:01.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:00.880+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:01.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:00.880+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:01.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:00.880+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:01.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:00.881+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:01.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:01.329+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:01.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:01.329+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:01.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:01.329+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:01.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:01.330+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:02.063 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:26:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:26:01.812+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 24597 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:26:02.063 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:01.847+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:02.064 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:01.847+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:02.064 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:01.847+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:02.064 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:01.848+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:02.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:02.059+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:02.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:02.059+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:02.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:02.059+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:02.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:02.060+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:02.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:02.284+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:02.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:02.284+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:02.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:02.284+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:02.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:02.285+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:03.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:02.864+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:03.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:02.864+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:03.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:02.864+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:03.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:02.865+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:03.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:03.088+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:03.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:03.088+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:03.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:03.088+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:03.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:03.089+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:03.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:03.299+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:03.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:03.299+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:03.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:03.299+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:03.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:03.300+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:04.117 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:03.862+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:04.117 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:03.862+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:04.118 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:03.862+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:04.118 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:03.863+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:04.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:04.113+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:04.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:04.113+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:04.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:04.113+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:04.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:04.115+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:04.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:04.306+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:04.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:04.306+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:04.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:04.306+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:04.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:04.307+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:05.081 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:04.814+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:05.081 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:04.814+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:05.081 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:04.814+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:05.081 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:04.814+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:05.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:05.308+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:05.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:05.308+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:05.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:05.308+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:05.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:05.310+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:05.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:05.077+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:05.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:05.077+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:05.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:05.077+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:05.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:05.078+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:06.108 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:05.824+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:06.108 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:05.824+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:06.109 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:05.824+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:06.109 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:05.825+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:06.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:06.343+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:06.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:06.343+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:06.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:06.343+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:06.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:06.344+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:06.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:06.104+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:06.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:06.104+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:06.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:06.104+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:06.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:06.105+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:07.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:26:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:26:06.815+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 24680 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:26:07.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:07.055+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:07.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:07.055+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:07.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:07.056+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:07.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:07.056+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:07.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:06.844+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:07.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:06.844+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:07.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:06.844+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:07.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:06.845+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:07.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:07.356+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:07.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:07.356+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:07.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:07.356+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:07.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:07.357+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:08.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:08.095+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:08.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:08.095+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:08.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:08.095+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:08.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:08.096+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:08.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:07.882+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:08.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:07.882+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:08.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:07.882+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:08.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:07.883+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:08.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:08.317+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:08.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:08.317+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:08.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:08.317+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:08.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:08.318+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:09.115 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:08.857+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:09.116 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:08.857+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:09.116 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:08.857+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:09.116 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:08.859+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:09.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:09.331+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:09.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:09.331+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:09.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:09.331+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:09.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:09.332+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:09.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:09.111+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:09.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:09.111+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:09.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:09.111+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:09.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:09.112+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:10.102 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:09.833+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:10.103 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:09.833+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:10.103 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:09.833+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:10.103 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:09.834+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:10.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:10.307+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:10.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:10.307+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:10.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:10.307+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:10.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:10.308+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:10.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:10.099+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:10.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:10.099+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:10.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:10.099+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:10.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:10.100+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:11.135 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:10.875+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:11.136 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:10.875+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:11.136 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:10.875+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:11.136 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:10.876+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:11.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:11.270+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:11.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:11.270+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:11.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:11.270+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:11.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:11.271+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:11.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:11.132+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:11.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:11.132+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:11.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:11.132+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:11.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:11.133+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:12.093 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:26:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:26:11.818+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 24764 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:26:12.094 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:11.917+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:12.094 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:11.917+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:12.094 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:11.917+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:12.094 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:11.918+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:12.094 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:12.089+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:12.094 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:12.089+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:12.095 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:12.089+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:12.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:12.090+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:12.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:12.299+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:12.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:12.299+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:12.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:12.299+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:12.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:12.300+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:13.318 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:12.934+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:13.319 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:12.934+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:13.319 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:12.934+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:13.319 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:12.935+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:13.319 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:13.104+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:13.320 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:13.104+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:13.320 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:13.104+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:13.320 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:13.104+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:13.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:13.315+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:13.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:13.315+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:13.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:13.315+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:13.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:13.316+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:14.282 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:14.097+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:14.283 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:14.097+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:14.283 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:14.097+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:14.283 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:14.098+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:14.283 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:13.974+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:14.283 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:13.974+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:14.284 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:13.974+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:14.284 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:13.975+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:14.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:14.279+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:14.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:14.279+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:14.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:14.279+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:14.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:14.280+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:15.279 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:15.124+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:15.279 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:15.124+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:15.280 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:15.124+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:15.280 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:15.125+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:15.280 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:14.964+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:15.280 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:14.964+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:15.280 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:14.964+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:15.280 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:14.965+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:15.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:15.275+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:15.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:15.275+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:15.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:15.275+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:15.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:15.276+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:16.283 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:15.964+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:16.283 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:15.964+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:16.283 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:15.964+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:16.283 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:15.965+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:16.283 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:16.100+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:16.284 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:16.100+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:16.284 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:16.100+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:16.284 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:16.101+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:16.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:16.279+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:16.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:16.279+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:16.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:16.279+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:16.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:16.280+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:17.140 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:26:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:26:16.820+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 24847 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:26:17.140 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:16.948+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:17.141 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:16.948+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:17.141 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:16.948+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:17.141 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:16.949+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:17.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:17.136+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:17.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:17.136+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:17.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:17.136+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:17.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:17.137+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:17.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:17.294+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:17.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:17.294+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:17.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:17.294+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:17.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:17.294+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:18.300 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:17.957+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:18.300 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:17.957+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:18.300 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:17.957+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:18.300 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:17.958+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:18.300 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:18.156+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:18.301 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:18.156+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:18.301 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:18.156+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:18.301 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:18.157+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:18.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:18.296+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:18.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:18.296+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:18.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:18.296+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:18.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:18.296+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:19.320 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:19.133+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:19.320 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:19.133+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:19.320 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:19.133+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:19.320 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:19.134+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:19.321 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:18.989+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:19.321 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:18.989+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:19.321 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:18.989+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:19.321 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:18.990+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:19.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:19.316+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:19.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:19.316+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:19.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:19.316+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:19.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:19.317+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:20.339 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:20.184+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:20.340 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:20.184+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:20.340 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:20.184+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:20.340 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:20.184+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:20.340 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:19.953+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:20.340 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:19.953+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:20.340 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:19.953+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:20.341 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:19.954+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:20.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:20.335+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:20.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:20.335+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:20.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:20.335+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:20.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:20.336+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:21.294 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:20.968+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:21.295 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:20.968+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:21.295 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:20.968+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:21.295 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:20.969+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:21.295 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:21.214+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:21.295 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:21.214+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:21.295 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:21.214+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:21.295 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:21.215+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:21.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:21.290+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:21.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:21.290+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:21.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:21.290+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:21.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:21.291+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:21.994 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:26:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:26:21.823+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 24931 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:26:22.288 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:22.177+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:22.288 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:22.177+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:22.288 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:22.177+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:22.289 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:22.178+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:22.289 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:21.990+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:22.289 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:21.990+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:22.289 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:21.990+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:22.289 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:21.991+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:22.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:22.283+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:22.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:22.283+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:22.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:22.283+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:22.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:22.284+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:23.275 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:22.976+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:23.275 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:22.976+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:23.275 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:22.976+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:23.275 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:22.977+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:23.275 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:23.139+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:23.275 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:23.139+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:23.276 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:23.139+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:23.276 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:23.140+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:23.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:23.271+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:23.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:23.271+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:23.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:23.271+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:23.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:23.272+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:24.275 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:23.996+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:24.275 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:23.996+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:24.276 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:23.996+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:24.276 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:23.997+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:24.276 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:24.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:24.276 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:24.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:24.276 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:24.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:24.277 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:24.171+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:24.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:24.271+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:24.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:24.271+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:24.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:24.271+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:24.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:24.273+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:25.250 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:25.129+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:25.250 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:25.129+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:25.250 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:25.129+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:25.251 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:25.131+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:25.251 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:24.965+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:25.251 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:24.965+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:25.251 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:24.965+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:25.251 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:24.965+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:25.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:25.246+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:25.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:25.246+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:25.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:25.246+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:25.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:25.247+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:26.297 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:25.926+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:26.298 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:25.926+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:26.298 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:25.926+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:26.298 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:25.927+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:26.298 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:26.122+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:26.298 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:26.122+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:26.298 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:26.122+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:26.299 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:26.122+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:26.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:26.294+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:26.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:26.294+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:26.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:26.294+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:26.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:26.295+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:27.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:26:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:26:26.825+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 25014 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:26:27.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:26.923+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:27.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:26.923+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:27.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:26.923+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:27.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:26.924+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:27.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:27.154+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:27.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:27.154+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:27.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:27.154+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:27.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:27.155+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:27.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:27.292+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:27.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:27.292+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:27.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:27.292+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:27.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:27.293+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:28.154 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:27.880+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:28.154 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:27.880+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:28.155 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:27.880+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:28.155 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:27.880+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:28.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:28.150+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:28.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:28.150+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:28.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:28.150+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:28.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:28.152+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:28.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:28.251+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:28.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:28.251+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:28.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:28.251+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:28.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:28.252+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:29.149 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:28.865+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:29.149 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:28.865+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:29.149 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:28.865+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:29.150 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:28.866+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:29.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:29.269+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:29.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:29.269+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:29.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:29.269+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:29.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:29.270+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:29.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:29.145+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:29.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:29.145+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:29.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:29.145+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:29.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:29.146+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:30.136 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:29.873+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:30.137 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:29.873+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:30.137 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:29.873+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:30.137 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:29.874+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:30.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:30.316+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:30.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:30.316+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:30.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:30.316+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:30.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:30.317+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:30.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:30.133+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:30.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:30.133+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:30.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:30.133+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:30.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:30.133+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:31.143 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:30.869+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:31.144 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:30.869+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:31.144 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:30.869+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:31.144 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:30.870+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:31.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:31.271+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:31.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:31.271+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:31.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:31.271+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:31.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:31.272+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:31.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:31.140+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:31.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:31.140+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:31.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:31.140+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:31.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:31.141+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:32.114 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:26:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:26:31.828+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 25098 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:26:32.114 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:31.900+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:32.115 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:31.900+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:32.115 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:31.900+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:32.115 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:31.901+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:32.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:32.110+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:32.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:32.110+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:32.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:32.110+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:32.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:32.111+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:32.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:32.233+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:32.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:32.233+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:32.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:32.233+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:32.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:32.235+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:32.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:32.853+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:32.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:32.853+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:32.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:32.853+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:32.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:32.853+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:33.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:33.249+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:33.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:33.249+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:33.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:33.249+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:33.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:33.249+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:33.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:33.136+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:33.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:33.136+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:33.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:33.136+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:33.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:33.137+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:34.128 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:33.850+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:34.128 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:33.850+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:34.128 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:33.850+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:34.128 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:33.851+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:34.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:34.124+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:34.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:34.124+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:34.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:34.124+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:34.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:34.125+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:34.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:34.276+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:34.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:34.276+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:34.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:34.276+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:34.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:34.277+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:35.150 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:34.894+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:35.150 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:34.894+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:35.150 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:34.894+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:35.150 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:34.895+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:35.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:35.323+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:35.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:35.323+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:35.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:35.323+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:35.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:35.324+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:35.146+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:35.146+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:35.146+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:35.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:35.147+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:36.294 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:36.138+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:36.294 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:36.138+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:36.294 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:36.138+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:36.295 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:36.139+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:36.295 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:35.931+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:36.295 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:35.931+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:36.295 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:35.931+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:36.295 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:35.932+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:36.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:36.290+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:36.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:36.290+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:36.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:36.290+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:36.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:36.291+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:37.108 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:26:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:26:36.830+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 25181 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:26:37.108 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:36.974+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:37.108 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:36.974+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:37.108 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:36.974+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:37.108 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:36.974+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:37.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:37.291+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:37.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:37.291+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:37.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:37.291+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:37.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:37.292+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:37.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:37.104+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:37.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:37.104+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:37.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:37.104+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:37.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:37.105+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:38.258 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:37.969+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:38.258 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:37.969+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:38.258 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:37.969+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:38.258 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:37.970+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:38.259 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:38.117+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:38.259 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:38.117+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:38.259 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:38.117+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:38.259 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:38.118+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:38.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:38.254+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:38.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:38.254+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:38.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:38.254+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:38.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:38.255+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:39.262 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:38.945+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:39.262 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:38.945+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:39.263 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:38.945+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:39.263 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:38.947+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:39.263 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:39.156+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:39.263 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:39.156+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:39.263 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:39.156+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:39.263 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:39.157+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:39.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:39.258+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:39.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:39.258+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:39.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:39.258+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:39.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:39.259+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:40.203 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:39.913+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:40.204 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:39.913+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:40.204 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:39.913+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:40.204 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:39.914+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:40.204 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:40.127+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:40.204 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:40.127+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:40.204 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:40.127+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:40.205 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:40.128+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:40.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:40.227+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:40.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:40.227+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:40.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:40.227+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:40.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:40.228+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:41.200 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:40.923+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:41.201 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:40.923+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:41.201 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:40.923+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:41.201 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:40.924+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:41.201 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:41.121+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:41.202 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:41.121+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:41.202 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:41.121+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:41.202 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:41.122+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:41.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:41.197+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:41.666 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:41.197+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:41.667 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:41.197+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:41.667 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:41.198+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:42.116 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:26:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:26:41.833+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 25265 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:26:42.117 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:41.964+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:42.117 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:41.964+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:42.117 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:41.964+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:42.117 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:41.965+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:42.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:42.172+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:42.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:42.172+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:42.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:42.172+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:42.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:42.173+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:42.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:42.113+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:42.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:42.113+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:42.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:42.113+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:42.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:42.114+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:43.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:43.010+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:43.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:43.010+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:43.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:43.010+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:43.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:43.011+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:43.078+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:43.078+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:43.078+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:43.079+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:43.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:43.203+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:43.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:43.203+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:43.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:43.203+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:43.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:43.204+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:44.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:44.163+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:44.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:44.163+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:44.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:44.163+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:44.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:44.164+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:44.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:44.096+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:44.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:44.096+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:44.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:44.096+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:44.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:44.097+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:44.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:43.975+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:44.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:43.975+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:44.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:43.975+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:44.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:43.976+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:45.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:45.155+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:45.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:45.155+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:45.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:45.155+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:45.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:45.156+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:45.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:45.059+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:45.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:45.059+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:45.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:45.059+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:45.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:45.060+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:45.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:44.956+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:45.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:44.956+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:45.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:44.956+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:45.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:44.956+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:46.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:46.174+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:46.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:46.174+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:46.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:46.174+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:46.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:46.175+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:46.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:46.099+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:46.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:46.099+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:46.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:46.099+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:46.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:46.099+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:46.410 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:45.975+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:46.410 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:45.975+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:46.410 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:45.975+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:46.410 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:45.976+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:47.109 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:26:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:26:46.836+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 25348 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:26:47.109 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:46.953+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:47.109 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:46.953+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:47.109 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:46.953+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:47.109 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:46.954+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:47.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:47.105+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:47.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:47.105+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:47.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:47.105+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:47.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:47.106+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:47.410 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:47.189+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:47.410 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:47.189+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:47.410 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:47.189+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:47.410 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:47.190+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:48.188 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:47.910+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:48.188 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:47.910+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:48.188 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:47.910+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:48.189 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:47.911+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:48.189 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:48.144+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:48.189 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:48.144+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:48.189 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:48.144+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:48.189 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:48.144+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:48.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:48.184+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:48.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:48.184+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:48.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:48.184+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:48.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:48.185+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:49.211 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:48.946+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:49.211 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:48.946+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:49.211 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:48.946+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:49.212 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:48.947+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:49.212 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:49.116+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:49.212 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:49.116+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:49.212 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:49.116+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:49.212 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:49.117+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:49.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:49.207+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:49.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:49.207+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:49.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:49.207+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:49.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:49.208+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:50.204 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:50.091+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:50.205 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:50.091+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:50.205 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:50.091+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:50.205 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:50.092+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:50.205 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:49.937+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:50.205 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:49.937+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:50.205 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:49.937+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:50.206 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:49.938+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:50.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:50.255+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:50.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:50.255+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:50.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:50.255+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:50.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:50.255+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:51.270 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:51.058+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:51.271 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:51.058+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:51.271 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:51.058+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:51.271 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:51.059+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:51.271 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:50.976+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:51.271 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:50.976+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:51.272 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:50.976+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:51.272 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:50.977+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:51.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:51.266+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:51.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:51.266+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:51.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:51.266+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:51.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:51.267+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:52.019 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:26:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:26:51.838+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 25432 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:26:52.020 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:51.930+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:52.020 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:51.930+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:52.020 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:51.930+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:52.020 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:51.930+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:52.293 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:52.016+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:52.294 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:52.016+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:52.294 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:52.016+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:52.294 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:52.016+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:52.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:52.289+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:52.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:52.289+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:52.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:52.289+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:52.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:52.290+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:53.305 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:52.921+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:53.306 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:52.921+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:53.306 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:52.921+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:53.306 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:52.921+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:53.306 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:52.998+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:53.306 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:52.998+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:53.306 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:52.998+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:53.306 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:52.999+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:53.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:53.301+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:53.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:53.301+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:53.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:53.301+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:53.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:53.303+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:54.332 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:53.985+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:54.332 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:53.985+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:54.332 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:53.985+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:54.332 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:53.986+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:54.332 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:53.949+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:54.333 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:53.949+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:54.333 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:53.949+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:54.333 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:53.950+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:54.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:54.328+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:54.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:54.328+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:54.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:54.328+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:54.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:54.329+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:55.325 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:55.010+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:55.326 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:55.010+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:55.326 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:55.010+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:55.326 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:55.012+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:55.326 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:54.908+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:55.326 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:54.908+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:55.327 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:54.908+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:55.327 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:54.908+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:55.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:55.322+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:55.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:55.322+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:55.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:55.322+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:55.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:55.323+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:56.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:55.882+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:56.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:55.882+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:56.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:55.882+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:56.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:55.882+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:56.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:55.997+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:56.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:55.997+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:56.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:55.997+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:56.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:55.998+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:56.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:56.312+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:56.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:56.312+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:56.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:56.312+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:56.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:56.313+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:57.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:26:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:26:56.841+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 25515 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:26:57.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:56.881+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:57.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:56.881+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:57.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:56.881+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:57.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:56.882+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:57.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:57.031+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:57.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:57.031+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:57.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:57.031+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:57.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:57.032+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:57.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:57.273+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:57.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:57.273+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:57.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:57.273+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:57.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:57.274+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:58.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:57.837+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:58.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:57.837+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:58.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:57.837+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:58.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:57.838+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:58.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:58.018+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:58.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:58.018+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:58.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:58.018+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:58.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:58.019+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:58.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:58.318+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:58.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:58.318+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:58.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:58.318+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:58.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:58.319+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:59.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:59.068+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:59.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:59.068+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:59.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:59.068+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:26:59.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:26:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:26:59.069+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:26:59.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:58.849+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:59.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:58.849+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:59.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:58.849+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:26:59.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:58.850+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:26:59.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:59.357+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:59.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:59.357+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:59.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:59.357+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:26:59.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:26:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:26:59.358+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:00.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:59.855+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:00.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:59.855+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:00.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:59.855+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:00.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:26:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:26:59.856+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:00.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:00.081+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:00.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:00.081+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:00.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:00.081+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:00.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:00.082+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:00.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:00.369+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:00.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:00.369+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:00.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:00.369+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:00.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:00.370+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:01.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:00.861+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:01.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:00.861+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:01.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:00.861+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:01.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:00.863+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:01.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:01.036+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:01.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:01.036+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:01.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:01.036+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:01.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:01.037+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:01.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:01.392+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:01.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:01.392+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:01.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:01.392+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:01.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:01.393+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:02.039 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:27:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:27:01.844+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 25599 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:27:02.040 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:01.857+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:02.040 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:01.857+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:02.040 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:01.857+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:02.040 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:01.858+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:02.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:02.035+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:02.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:02.035+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:02.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:02.035+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:02.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:02.036+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:02.859 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:02.415+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:02.859 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:02.415+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:02.859 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:02.415+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:02.859 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:02.416+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:03.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:02.855+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:03.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:02.855+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:03.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:02.855+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:03.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:02.856+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:03.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:03.066+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:03.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:03.066+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:03.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:03.066+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:03.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:03.067+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:03.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:03.457+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:03.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:03.457+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:03.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:03.457+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:03.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:03.458+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:04.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:03.854+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:04.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:03.854+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:04.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:03.854+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:04.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:03.855+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:04.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:04.086+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:04.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:04.086+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:04.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:04.086+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:04.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:04.087+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:04.899 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:04.503+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:04.899 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:04.503+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:04.900 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:04.503+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:04.900 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:04.504+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:05.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:05.119+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:05.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:05.119+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:05.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:05.119+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:05.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:05.120+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:05.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:04.895+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:05.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:04.895+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:05.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:04.895+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:05.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:04.896+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:05.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:05.530+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:05.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:05.530+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:05.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:05.530+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:05.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:05.530+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:06.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:05.917+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:06.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:05.917+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:06.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:05.917+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:06.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:05.918+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:06.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:06.111+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:06.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:06.111+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:06.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:06.111+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:06.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:06.112+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:06.850 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:06.504+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:06.850 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:06.504+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:06.851 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:06.504+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:06.851 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:06.504+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:07.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:27:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:27:06.846+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 25682 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:27:07.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:06.945+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:07.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:06.945+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:07.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:06.945+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:07.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:06.946+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:07.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:07.091+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:07.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:07.091+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:07.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:07.091+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:07.159 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:07.092+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:07.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:07.539+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:07.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:07.539+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:07.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:07.539+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:07.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:07.540+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:08.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:07.953+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:08.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:07.953+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:08.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:07.953+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:08.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:07.954+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:08.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:08.084+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:08.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:08.084+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:08.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:08.084+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:08.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:08.085+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:08.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:08.539+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:08.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:08.539+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:08.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:08.539+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:08.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:08.539+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:09.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:09.051+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:09.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:09.051+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:09.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:09.051+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:09.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:09.052+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:09.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:08.993+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:09.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:08.993+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:09.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:08.993+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:09.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:08.994+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:09.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:09.585+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:09.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:09.585+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:09.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:09.585+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:09.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:09.586+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:10.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:10.053+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:10.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:10.053+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:10.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:10.053+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:10.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:10.053+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:10.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:10.032+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:10.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:10.032+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:10.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:10.032+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:10.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:10.032+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:10.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:10.561+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:10.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:10.561+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:10.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:10.561+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:10.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:10.561+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:11.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:11.013+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:11.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:11.013+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:11.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:11.013+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:11.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:11.014+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:11.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:11.040+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:11.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:11.040+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:11.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:11.040+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:11.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:11.041+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:11.853 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:11.514+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:11.853 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:11.514+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:11.854 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:11.514+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:11.854 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:11.515+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:12.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:27:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:27:11.849+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 25766 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:27:12.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:12.085+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:12.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:12.085+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:12.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:12.085+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:12.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:12.086+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:12.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:11.996+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:12.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:11.996+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:12.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:11.996+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:12.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:11.996+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:12.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:12.524+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:12.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:12.524+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:12.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:12.524+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:12.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:12.526+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:13.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:13.036+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:13.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:13.036+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:13.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:13.036+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:13.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:13.037+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:13.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:13.131+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:13.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:13.131+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:13.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:13.131+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:13.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:13.132+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:13.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:13.534+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:13.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:13.534+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:13.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:13.534+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:13.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:13.535+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:14.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:14.057+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:14.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:14.057+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:14.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:14.057+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:14.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:14.058+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:14.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:14.129+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:14.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:14.129+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:14.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:14.129+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:14.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:14.130+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:14.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:14.500+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:14.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:14.500+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:14.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:14.500+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:14.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:14.501+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:15.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:15.063+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:15.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:15.063+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:15.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:15.063+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:15.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:15.064+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:15.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:15.163+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:15.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:15.163+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:15.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:15.164+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:15.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:15.165+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:15.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:15.471+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:15.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:15.471+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:15.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:15.471+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:15.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:15.471+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:16.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:16.058+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:16.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:16.058+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:16.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:16.058+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:16.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:16.059+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:16.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:16.127+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:16.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:16.127+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:16.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:16.127+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:16.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:16.128+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:16.855 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:16.471+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:16.856 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:16.471+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:16.856 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:16.471+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:16.856 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:16.472+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:17.155 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:27:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:27:16.851+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 25849 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:27:17.156 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:17.082+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:17.156 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:17.082+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:17.156 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:17.082+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:17.156 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:17.082+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:17.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:17.151+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:17.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:17.151+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:17.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:17.151+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:17.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:17.152+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:17.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:17.501+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:17.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:17.501+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:17.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:17.501+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:17.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:17.502+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:18.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:18.129+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:18.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:18.129+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:18.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:18.129+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:18.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:18.130+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:18.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:18.126+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:18.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:18.126+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:18.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:18.126+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:18.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:18.126+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:18.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:18.502+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:18.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:18.502+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:18.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:18.502+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:18.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:18.503+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:19.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:19.105+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:19.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:19.105+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:19.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:19.105+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:19.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:19.106+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:19.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:19.103+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:19.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:19.103+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:19.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:19.103+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:19.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:19.104+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:19.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:19.474+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:19.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:19.474+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:19.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:19.474+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:19.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:19.474+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:20.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:20.075+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:20.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:20.075+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:20.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:20.075+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:20.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:20.075+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:20.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:20.063+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:20.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:20.063+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:20.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:20.063+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:20.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:20.064+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:20.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:20.483+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:20.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:20.483+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:20.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:20.483+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:20.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:20.484+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:21.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:21.056+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:21.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:21.056+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:21.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:21.056+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:21.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:21.056+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:21.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:21.092+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:21.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:21.092+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:21.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:21.092+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:21.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:21.092+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:21.806 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:21.491+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:21.807 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:21.491+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:21.807 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:21.491+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:21.807 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:21.492+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:22.092 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:27:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:27:21.854+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 25933 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:27:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:22.088+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:22.088+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:22.088+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:22.089+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:22.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:22.095+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:22.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:22.095+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:22.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:22.095+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:22.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:22.096+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:22.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:22.518+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:22.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:22.518+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:22.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:22.518+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:22.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:22.519+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:23.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:23.120+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:23.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:23.120+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:23.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:23.120+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:23.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:23.122+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:23.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:23.088+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:23.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:23.088+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:23.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:23.088+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:23.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:23.089+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:23.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:23.470+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:23.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:23.470+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:23.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:23.470+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:23.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:23.471+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:24.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:24.118+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:24.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:24.118+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:24.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:24.118+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:24.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:24.118+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:24.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:24.116+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:24.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:24.116+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:24.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:24.116+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:24.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:24.116+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:24.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:24.518+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:24.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:24.518+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:24.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:24.518+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:24.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:24.519+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:25.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:25.082+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:25.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:25.082+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:25.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:25.082+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:25.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:25.082+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:25.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:25.167+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:25.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:25.167+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:25.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:25.167+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:25.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:25.168+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:25.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:25.533+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:25.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:25.533+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:25.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:25.533+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:25.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:25.534+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:26.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:26.178+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:26.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:26.178+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:26.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:26.178+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:26.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:26.179+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:26.086+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:26.086+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:26.086+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:26.087+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:26.861 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:26.487+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:26.861 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:26.487+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:26.861 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:26.487+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:26.861 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:26.488+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:27.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:27:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:27:26.857+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 26016 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:27:27.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:27.099+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:27.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:27.099+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:27.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:27.099+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:27.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:27.100+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:27.517 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:27.191+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:27.517 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:27.191+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:27.517 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:27.191+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:27.517 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:27.192+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:27.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:27.513+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:27.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:27.513+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:27.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:27.513+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:27.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:27.514+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:28.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:28.105+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:28.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:28.105+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:28.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:28.105+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:28.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:28.106+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:28.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:28.177+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:28.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:28.177+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:28.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:28.177+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:28.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:28.178+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:28.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:28.482+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:28.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:28.482+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:28.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:28.482+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:28.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:28.482+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:29.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:29.149+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:29.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:29.149+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:29.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:29.149+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:29.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:29.150+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:29.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:29.138+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:29.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:29.138+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:29.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:29.138+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:29.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:29.139+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:29.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:29.451+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:29.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:29.451+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:29.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:29.451+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:29.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:29.451+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:30.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:30.133+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:30.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:30.133+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:30.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:30.133+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:30.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:30.134+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:30.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:30.143+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:30.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:30.143+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:30.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:30.143+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:30.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:30.143+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:30.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:30.424+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:30.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:30.424+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:30.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:30.424+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:30.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:30.425+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:31.124 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:31.118+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:31.124 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:31.118+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:31.124 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:31.118+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:31.395 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:31.119+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:31.396 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:31.166+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:31.396 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:31.166+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:31.396 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:31.166+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:31.396 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:31.167+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:31.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:31.392+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:31.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:31.392+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:31.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:31.392+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:31.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:31.393+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:32.133 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:27:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:27:31.860+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 26100 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:27:32.394 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:32.129+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:32.395 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:32.129+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:32.395 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:32.129+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:32.395 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:32.130+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:32.396 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:32.130+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:32.396 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:32.130+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:32.396 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:32.130+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:32.396 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:32.131+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:32.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:32.390+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:32.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:32.390+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:32.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:32.390+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:32.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:32.391+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:33.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:33.149+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:33.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:33.149+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:33.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:33.149+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:33.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:33.150+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:33.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:33.124+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:33.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:33.124+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:33.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:33.124+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:33.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:33.124+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:33.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:33.437+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:33.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:33.437+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:33.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:33.437+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:33.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:33.438+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:34.406 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:34.169+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:34.406 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:34.169+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:34.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:34.169+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:34.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:34.170+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:34.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:34.158+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:34.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:34.158+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:34.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:34.158+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:34.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:34.159+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:34.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:34.402+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:34.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:34.402+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:34.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:34.402+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:34.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:34.403+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:35.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:35.370+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:35.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:35.370+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:35.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:35.370+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:35.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:35.371+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:35.134+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:35.134+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:35.134+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:35.135+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:35.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:35.176+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:35.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:35.176+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:35.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:35.176+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:35.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:35.177+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:36.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:36.341+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:36.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:36.341+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:36.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:36.341+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:36.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:36.341+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:36.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:36.169+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:36.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:36.169+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:36.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:36.169+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:36.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:36.170+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:36.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:36.172+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:36.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:36.172+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:36.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:36.172+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:36.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:36.173+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:37.143 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:27:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:27:36.863+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 26182 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:27:37.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:37.139+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:37.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:37.139+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:37.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:37.139+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:37.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:37.140+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:37.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:37.371+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:37.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:37.371+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:37.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:37.371+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:37.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:37.372+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:37.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:37.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:37.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:37.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:37.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:37.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:37.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:37.171+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:38.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:38.210+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:38.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:38.210+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:38.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:38.210+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:38.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:38.211+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:38.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:38.123+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:38.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:38.123+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:38.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:38.123+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:38.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:38.124+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:38.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:38.412+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:38.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:38.412+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:38.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:38.412+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:38.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:38.412+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:39.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:39.158+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:39.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:39.158+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:39.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:39.158+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:39.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:39.159+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:39.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:39.379+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:39.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:39.379+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:39.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:39.379+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:39.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:39.380+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:39.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:39.225+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:39.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:39.225+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:39.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:39.225+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:39.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:39.227+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:40.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:40.176+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:40.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:40.176+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:40.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:40.176+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:40.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:40.177+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:40.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:40.384+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:40.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:40.384+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:40.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:40.384+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:40.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:40.385+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:40.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:40.272+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:40.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:40.272+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:40.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:40.272+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:40.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:40.272+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:41.435 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:41.169+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:41.436 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:41.169+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:41.436 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:41.169+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:41.436 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:41.170+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:41.436 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:41.307+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:41.436 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:41.307+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:41.437 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:41.307+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:41.437 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:41.308+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:41.742 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:41.432+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:41.742 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:41.432+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:41.742 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:41.432+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:41.742 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:41.433+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:41.994 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:27:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:27:41.865+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 26265 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:27:42.318 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:42.130+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:42.318 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:42.130+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:42.318 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:42.130+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:42.319 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:42.130+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:42.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:42.314+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:42.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:42.314+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:42.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:42.314+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:42.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:42.315+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:42.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:42.387+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:42.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:42.387+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:42.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:42.387+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:42.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:42.388+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:43.498 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:43.350+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:43.498 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:43.350+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:43.498 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:43.350+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:43.498 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:43.351+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:43.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:43.165+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:43.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:43.165+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:43.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:43.165+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:43.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:43.166+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:43.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:43.296+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:43.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:43.296+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:43.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:43.296+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:43.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:43.296+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:44.382 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:44.275+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:44.383 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:44.275+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:44.383 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:44.275+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:44.383 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:44.276+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:44.383 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:44.128+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:44.383 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:44.128+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:44.383 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:44.128+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:44.384 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:44.129+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:44.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:44.378+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:44.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:44.378+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:44.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:44.378+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:44.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:44.379+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:45.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:45.414+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:45.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:45.414+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:45.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:45.414+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:45.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:45.415+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:45.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:45.171+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:45.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:45.171+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:45.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:45.171+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:45.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:45.172+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:45.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:45.254+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:45.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:45.254+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:45.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:45.254+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:45.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:45.255+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:46.394 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:46.239+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:46.394 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:46.239+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:46.394 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:46.239+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:46.395 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:46.240+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:46.395 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:46.128+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:46.395 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:46.128+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:46.395 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:46.128+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:46.395 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:46.129+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:46.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:46.390+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:46.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:46.390+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:46.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:46.390+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:46.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:46.391+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:47.128 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:27:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:27:46.868+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 26347 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:27:47.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:47.289+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:47.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:47.289+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:47.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:47.289+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:47.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:47.290+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:47.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:47.124+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:47.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:47.124+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:47.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:47.124+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:47.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:47.125+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:47.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:47.411+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:47.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:47.411+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:47.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:47.411+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:47.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:47.412+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:48.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:48.124+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:48.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:48.124+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:48.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:48.124+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:48.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:48.124+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:48.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:48.281+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:48.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:48.281+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:48.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:48.281+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:48.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:48.281+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:48.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:48.461+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:48.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:48.461+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:48.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:48.461+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:48.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:48.462+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:49.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:49.239+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:49.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:49.239+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:49.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:49.239+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:49.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:49.240+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:49.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:49.134+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:49.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:49.134+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:49.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:49.134+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:49.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:49.135+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:49.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:49.498+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:49.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:49.498+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:49.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:49.498+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:49.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:49.498+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:50.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:50.114+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:50.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:50.114+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:50.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:50.114+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:50.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:50.115+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:50.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:50.241+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:50.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:50.241+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:50.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:50.241+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:50.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:50.242+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:50.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:50.458+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:50.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:50.458+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:50.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:50.458+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:50.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:50.459+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:51.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:51.192+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:51.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:51.192+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:51.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:51.192+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:51.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:51.193+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:51.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:51.068+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:51.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:51.068+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:51.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:51.068+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:51.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:51.069+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:51.743 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:51.462+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:51.743 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:51.462+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:51.743 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:51.462+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:51.743 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:51.463+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:52.000 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:27:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:27:51.871+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 26431 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:27:52.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:52.168+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:52.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:52.168+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:52.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:52.168+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:52.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:52.169+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:52.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:52.112+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:52.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:52.112+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:52.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:52.112+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:52.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:52.113+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:52.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:52.414+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:52.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:52.414+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:52.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:52.414+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:52.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:52.415+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:53.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:53.139+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:53.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:53.139+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:53.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:53.139+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:53.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:53.140+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:53.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:53.124+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:53.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:53.124+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:53.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:53.124+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:53.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:53.125+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:53.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:53.425+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:53.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:53.425+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:53.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:53.425+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:53.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:53.426+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:54.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:54.088+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:54.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:54.088+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:54.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:54.088+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:54.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:54.089+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:54.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:54.116+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:54.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:54.116+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:54.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:54.116+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:54.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:54.117+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:54.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:54.405+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:54.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:54.405+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:54.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:54.405+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:54.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:54.406+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:55.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:55.075+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:55.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:55.075+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:55.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:55.075+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:55.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:55.075+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:55.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:55.125+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:55.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:55.125+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:55.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:55.125+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:55.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:55.125+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:55.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:55.416+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:55.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:55.416+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:55.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:55.416+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:55.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:55.417+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:56.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:56.042+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:56.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:56.042+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:56.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:56.042+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:56.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:56.043+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:56.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:56.152+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:56.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:56.152+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:56.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:56.152+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:56.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:56.153+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:56.878 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:56.424+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:56.878 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:56.424+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:56.878 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:56.424+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:56.878 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:56.425+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:57.152 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:27:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:27:56.874+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 26513 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:27:57.152 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:57.093+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:57.152 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:57.093+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:57.153 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:57.093+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:57.153 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:57.094+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:57.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:57.148+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:57.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:57.148+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:57.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:57.148+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:57.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:57.149+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:57.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:57.421+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:57.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:57.421+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:57.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:57.421+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:57.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:57.422+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:58.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:58.169+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:58.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:58.169+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:58.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:58.169+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:58.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:58.170+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:58.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:58.054+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:58.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:58.054+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:58.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:58.054+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:58.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:58.055+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:58.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:58.464+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:58.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:58.464+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:58.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:58.464+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:58.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:58.465+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:59.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:59.131+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:59.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:59.131+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:59.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:59.131+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:27:59.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:27:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:27:59.132+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:27:59.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:59.014+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:59.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:59.014+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:59.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:59.014+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:27:59.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:27:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:27:59.015+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:27:59.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:59.422+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:59.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:59.422+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:59.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:59.422+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:27:59.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:27:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:27:59.423+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:00.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:00.178+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:00.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:00.178+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:00.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:00.178+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:00.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:00.179+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:00.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:00.009+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:00.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:00.009+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:00.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:00.009+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:00.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:00.009+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:00.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:00.428+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:00.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:00.428+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:00.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:00.428+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:00.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:00.429+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:01.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:01.142+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:01.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:01.142+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:01.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:01.142+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:01.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:01.143+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:01.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:01.034+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:01.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:01.034+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:01.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:01.034+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:01.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:01.035+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:01.743 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:01.450+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:01.743 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:01.450+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:01.743 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:01.450+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:01.743 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:01.450+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:01.996 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:28:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:28:01.877+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 26598 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:28:02.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:02.168+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:02.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:02.168+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:02.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:02.168+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:02.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:02.168+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:02.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:01.992+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:02.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:01.992+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:02.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:01.992+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:02.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:01.994+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:02.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:02.470+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:02.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:02.470+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:02.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:02.470+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:02.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:02.471+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:03.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:02.988+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:03.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:02.988+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:03.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:02.988+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:03.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:02.989+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:03.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:03.122+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:03.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:03.122+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:03.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:03.122+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:03.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:03.123+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:03.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:03.481+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:03.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:03.481+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:03.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:03.481+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:03.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:03.482+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:04.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:04.092+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:04.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:04.092+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:04.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:04.092+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:04.092+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:04.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:03.980+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:04.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:03.980+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:04.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:03.980+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:04.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:03.981+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:04.814 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:04.506+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:04.814 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:04.506+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:04.814 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:04.506+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:04.814 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:04.507+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:05.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:05.051+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:05.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:05.051+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:05.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:05.051+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:05.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:05.052+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:05.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:05.021+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:05.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:05.021+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:05.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:05.021+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:05.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:05.022+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:05.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:05.531+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:05.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:05.531+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:05.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:05.531+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:05.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:05.532+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:06.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:06.090+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:06.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:06.090+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:06.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:06.090+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:06.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:06.091+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:06.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:05.979+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:06.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:05.979+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:06.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:05.979+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:06.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:05.980+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:06.885 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:06.520+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:06.885 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:06.520+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:06.885 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:06.520+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:06.886 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:06.521+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:07.138 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:28:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:28:06.880+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 26680 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:28:07.138 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:07.002+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:07.139 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:07.002+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:07.139 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:07.002+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:07.139 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:07.003+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:07.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:07.134+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:07.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:07.134+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:07.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:07.134+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:07.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:07.135+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:07.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:07.537+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:07.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:07.537+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:07.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:07.537+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:07.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:07.538+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:08.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:08.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:08.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:08.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:08.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:08.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:08.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:08.171+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:08.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:07.969+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:08.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:07.969+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:08.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:07.969+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:08.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:07.970+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:08.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:08.501+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:08.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:08.501+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:08.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:08.501+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:08.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:08.502+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:09.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:09.129+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:09.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:09.129+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:09.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:09.129+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:09.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:09.130+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:09.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:09.002+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:09.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:09.002+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:09.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:09.002+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:09.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:09.002+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:09.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:09.497+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:09.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:09.497+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:09.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:09.497+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:09.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:09.497+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:10.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:10.016+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:10.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:10.016+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:10.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:10.016+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:10.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:10.017+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:10.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:10.136+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:10.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:10.136+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:10.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:10.136+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:10.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:10.137+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:10.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:10.524+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:10.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:10.524+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:10.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:10.524+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:10.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:10.525+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:11.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:11.031+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:11.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:11.031+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:11.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:11.031+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:11.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:11.032+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:11.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:11.166+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:11.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:11.166+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:11.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:11.166+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:11.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:11.167+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:11.886 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:11.564+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:11.886 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:11.564+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:11.887 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:11.564+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:11.887 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:11.564+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:12.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:28:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:28:11.882+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 26765 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:28:12.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:12.071+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:12.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:12.071+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:12.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:12.071+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:12.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:12.071+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:12.520 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:12.172+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:12.520 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:12.172+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:12.520 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:12.172+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:12.520 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:12.173+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:12.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:12.516+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:12.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:12.516+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:12.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:12.516+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:12.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:12.517+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:13.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:13.083+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:13.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:13.083+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:13.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:13.083+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:13.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:13.084+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:13.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:13.178+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:13.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:13.178+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:13.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:13.178+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:13.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:13.179+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:13.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:13.552+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:13.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:13.552+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:13.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:13.552+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:13.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:13.553+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:14.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:14.176+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:14.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:14.176+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:14.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:14.176+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:14.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:14.177+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:14.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:14.099+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:14.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:14.100+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:14.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:14.100+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:14.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:14.101+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:14.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:14.526+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:14.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:14.526+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:14.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:14.526+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:14.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:14.527+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:15.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:15.225+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:15.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:15.225+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:15.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:15.225+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:15.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:15.226+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:15.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:15.124+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:15.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:15.124+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:15.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:15.124+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:15.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:15.125+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:15.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:15.571+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:15.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:15.571+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:15.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:15.571+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:15.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:15.571+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:16.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:16.220+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:16.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:16.220+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:16.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:16.220+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:16.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:16.221+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:16.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:16.133+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:16.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:16.133+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:16.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:16.133+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:16.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:16.134+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:16.889 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:16.593+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:16.890 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:16.593+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:16.890 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:16.593+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:16.890 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:16.594+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:17.145 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:28:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:28:16.885+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 26847 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:28:17.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:17.141+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:17.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:17.141+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:17.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:17.141+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:17.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:17.142+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:17.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:17.242+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:17.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:17.242+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:17.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:17.242+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:17.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:17.242+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:17.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:17.556+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:17.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:17.556+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:17.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:17.556+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:17.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:17.557+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:18.561 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:18.186+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:18.561 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:18.186+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:18.561 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:18.186+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:18.561 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:18.186+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:18.561 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:18.206+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:18.562 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:18.206+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:18.562 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:18.206+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:18.562 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:18.208+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:18.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:18.557+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:18.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:18.557+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:18.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:18.557+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:18.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:18.558+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:19.535 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:19.179+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:19.535 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:19.179+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:19.535 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:19.179+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:19.535 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:19.180+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:19.536 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:19.207+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:19.536 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:19.208+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:19.536 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:19.208+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:19.536 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:19.208+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:19.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:19.531+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:19.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:19.531+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:19.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:19.531+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:19.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:19.532+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:20.543 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:20.217+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:20.544 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:20.217+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:20.544 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:20.217+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:20.544 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:20.218+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:20.545 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:20.222+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:20.545 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:20.222+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:20.545 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:20.222+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:20.545 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:20.223+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:20.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:20.540+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:20.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:20.540+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:20.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:20.540+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:20.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:20.541+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:21.496 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:21.269+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:21.496 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:21.269+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:21.496 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:21.269+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:21.496 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:21.270+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:21.497 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:21.221+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:21.497 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:21.221+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:21.497 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:21.221+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:21.497 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:21.222+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:21.892 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:21.492+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:21.892 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:21.492+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:21.892 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:21.492+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:21.893 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:21.493+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:22.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:28:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:28:21.888+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 26932 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:28:22.538 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:22.280+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:22.539 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:22.280+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:22.539 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:22.280+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:22.539 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:22.281+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:22.539 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:22.254+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:22.539 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:22.254+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:22.539 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:22.254+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:22.540 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:22.254+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:22.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:22.534+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:22.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:22.534+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:22.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:22.534+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:22.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:22.535+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:23.498 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:23.277+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:23.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:23.277+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:23.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:23.277+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:23.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:23.278+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:23.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:23.238+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:23.499 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:23.238+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:23.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:23.238+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:23.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:23.238+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:23.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:23.502+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:23.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:23.502+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:23.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:23.502+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:23.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:23.502+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:24.529 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:24.322+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:24.529 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:24.322+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:24.529 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:24.322+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:24.530 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:24.324+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:24.530 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:24.229+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:24.530 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:24.229+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:24.530 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:24.229+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:24.530 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:24.230+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:24.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:24.525+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:24.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:24.525+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:24.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:24.525+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:24.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:24.526+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:25.569 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:25.261+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:25.570 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:25.261+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:25.570 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:25.261+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:25.570 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:25.262+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:25.570 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:25.309+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:25.570 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:25.309+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:25.571 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:25.309+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:25.571 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:25.310+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:25.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:25.565+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:25.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:25.565+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:25.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:25.565+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:25.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:25.566+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:26.601 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:26.269+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:26.601 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:26.269+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:26.601 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:26.269+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:26.601 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:26.270+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:26.602 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:26.325+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:26.602 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:26.325+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:26.602 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:26.325+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:26.602 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:26.326+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:26.896 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:26.597+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:26.896 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:26.597+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:26.896 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:26.597+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:26.896 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:26.598+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:27.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:28:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:28:26.891+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 27014 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:28:27.621 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:27.330+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:27.621 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:27.330+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:27.621 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:27.330+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:27.621 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:27.331+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:27.622 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:27.295+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:27.622 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:27.295+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:27.622 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:27.295+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:27.622 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:27.296+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:27.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:27.617+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:27.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:27.617+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:27.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:27.617+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:27.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:27.618+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:28.578 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:28.304+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:28.578 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:28.304+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:28.578 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:28.304+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:28.578 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:28.305+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:28.578 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:28.311+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:28.579 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:28.311+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:28.579 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:28.311+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:28.579 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:28.313+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:28.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:28.574+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:28.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:28.574+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:28.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:28.574+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:28.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:28.575+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:29.622 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:29.348+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:29.622 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:29.348+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:29.623 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:29.348+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:29.623 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:29.349+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:29.623 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:29.264+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:29.623 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:29.264+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:29.623 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:29.264+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:29.623 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:29.265+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:29.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:29.618+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:29.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:29.618+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:29.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:29.618+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:29.909 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:29.619+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:30.606 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:30.392+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:30.606 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:30.392+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:30.606 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:30.392+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:30.606 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:30.392+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:30.607 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:30.242+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:30.607 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:30.242+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:30.607 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:30.242+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:30.607 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:30.243+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:30.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:30.602+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:30.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:30.602+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:30.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:30.602+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:30.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:30.603+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:31.573 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:31.246+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:31.573 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:31.246+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:31.573 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:31.246+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:31.573 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:31.247+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:31.574 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:31.344+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:31.574 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:31.344+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:31.574 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:31.344+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:31.574 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:31.345+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:31.898 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:31.569+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:31.898 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:31.569+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:31.898 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:31.569+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:31.899 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:31.570+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:32.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:28:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:28:31.894+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 27099 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:28:32.535 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:32.249+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:32.536 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:32.249+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:32.536 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:32.249+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:32.536 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:32.250+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:32.536 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:32.316+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:32.536 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:32.316+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:32.536 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:32.316+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:32.537 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:32.317+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:32.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:32.531+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:32.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:32.531+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:32.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:32.531+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:32.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:32.533+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:33.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:33.347+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:33.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:33.347+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:33.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:33.347+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:33.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:33.348+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:33.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:33.493+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:33.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:33.493+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:33.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:33.493+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:33.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:33.493+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:33.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:33.247+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:33.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:33.247+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:33.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:33.247+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:33.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:33.248+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:34.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:34.385+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:34.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:34.385+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:34.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:34.385+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:34.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:34.387+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:34.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:34.466+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:34.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:34.466+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:34.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:34.466+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:34.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:34.467+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:34.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:34.252+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:34.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:34.252+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:34.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:34.252+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:34.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:34.253+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:35.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:35.452+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:35.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:35.452+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:35.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:35.452+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:35.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:35.453+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:35.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:35.215+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:35.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:35.215+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:35.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:35.215+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:35.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:35.216+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:35.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:35.354+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:35.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:35.354+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:35.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:35.354+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:35.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:35.355+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:36.462 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:36.173+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:36.463 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:36.173+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:36.463 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:36.173+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:36.463 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:36.174+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:36.463 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:36.318+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:36.463 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:36.318+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:36.464 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:36.318+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:36.464 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:36.319+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:36.901 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:36.459+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:36.902 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:36.459+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:36.902 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:36.459+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:36.902 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:36.460+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:37.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:28:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:28:36.897+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 27181 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:28:37.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:37.443+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:37.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:37.443+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:37.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:37.443+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:37.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:37.444+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:37.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:37.303+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:37.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:37.303+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:37.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:37.303+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:37.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:37.304+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:37.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:37.210+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:37.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:37.210+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:37.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:37.210+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:37.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:37.211+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:38.465 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:38.279+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:38.465 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:38.279+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:38.465 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:38.279+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:38.466 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:38.280+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:38.466 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:38.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:38.466 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:38.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:38.466 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:38.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:38.466 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:38.171+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:38.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:38.461+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:38.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:38.461+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:38.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:38.461+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:38.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:38.462+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:39.491 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:39.267+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:39.491 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:39.267+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:39.491 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:39.267+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:39.491 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:39.268+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:39.491 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:39.212+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:39.492 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:39.212+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:39.492 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:39.212+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:39.492 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:39.212+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:39.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:39.487+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:39.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:39.487+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:39.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:39.487+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:39.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:39.488+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:40.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:40.315+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:40.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:40.315+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:40.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:40.315+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:40.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:40.316+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:40.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:40.439+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:40.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:40.439+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:40.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:40.439+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:40.660 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:40.440+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:40.660 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:40.201+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:40.660 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:40.202+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:40.661 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:40.202+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:40.661 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:40.202+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:41.466 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:41.161+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:41.466 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:41.161+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:41.466 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:41.161+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:41.466 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:41.162+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:41.467 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:41.267+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:41.467 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:41.267+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:41.467 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:41.267+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:41.467 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:41.268+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:41.745 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:41.462+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:41.745 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:41.462+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:41.745 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:41.462+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:41.745 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:41.463+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:42.025 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:28:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:28:41.900+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 27266 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:28:42.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:42.251+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:42.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:42.251+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:42.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:42.251+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:42.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:42.252+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:42.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:42.182+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:42.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:42.182+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:42.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:42.182+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:42.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:42.183+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:42.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:42.449+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:42.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:42.449+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:42.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:42.449+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:42.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:42.450+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:43.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:43.403+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:43.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:43.403+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:43.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:43.403+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:43.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:43.404+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:43.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:43.165+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:43.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:43.166+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:43.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:43.166+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:43.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:43.166+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:43.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:43.210+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:43.501 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:43.210+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:43.501 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:43.211+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:43.501 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:43.211+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:44.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:44.251+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:44.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:44.251+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:44.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:44.251+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:44.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:44.252+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:44.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:44.126+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:44.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:44.126+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:44.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:44.126+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:44.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:44.127+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:44.750 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:44.445+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:44.750 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:44.445+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:44.750 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:44.445+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:44.750 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:44.446+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:45.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:45.224+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:45.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:45.224+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:45.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:45.224+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:45.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:45.225+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:45.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:45.131+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:45.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:45.131+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:45.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:45.131+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:45.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:45.132+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:45.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:45.416+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:45.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:45.416+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:45.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:45.416+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:45.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:45.417+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:46.394 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:46.092+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:46.394 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:46.092+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:46.394 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:46.092+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:46.395 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:46.093+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:46.395 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:46.243+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:46.395 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:46.243+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:46.395 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:46.243+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:46.395 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:46.243+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:46.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:46.390+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:46.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:46.390+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:46.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:46.390+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:46.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:46.391+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:47.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:28:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:28:46.903+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 27348 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:28:47.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:47.116+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:47.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:47.116+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:47.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:47.116+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:47.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:47.117+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:47.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:47.364+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:47.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:47.364+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:47.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:47.364+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:47.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:47.365+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:47.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:47.250+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:47.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:47.250+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:47.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:47.250+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:47.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:47.251+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:48.395 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:48.098+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:48.395 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:48.098+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:48.396 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:48.098+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:48.396 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:48.099+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:48.396 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:48.213+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:48.396 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:48.213+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:48.396 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:48.213+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:48.396 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:48.214+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:48.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:48.391+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:48.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:48.391+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:48.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:48.391+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:48.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:48.392+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:49.396 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:49.258+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:49.396 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:49.258+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:49.397 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:49.258+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:49.397 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:49.259+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:49.397 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:49.133+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:49.397 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:49.133+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:49.398 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:49.133+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:49.398 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:49.134+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:49.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:49.392+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:49.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:49.392+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:49.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:49.392+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:49.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:49.393+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:50.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:50.355+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:50.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:50.355+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:50.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:50.355+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:50.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:50.356+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:50.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:50.213+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:50.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:50.213+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:50.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:50.213+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:50.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:50.214+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:50.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:50.134+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:50.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:50.134+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:50.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:50.134+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:50.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:50.135+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:51.161 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:51.112+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:51.161 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:51.112+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:51.161 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:51.112+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:51.161 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:51.113+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:51.531 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:51.403+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:51.531 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:51.403+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:51.531 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:51.403+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:51.531 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:51.404+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:51.531 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:51.185+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:51.532 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:51.185+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:51.532 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:51.185+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:51.532 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:51.186+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:52.168 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:28:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:28:51.905+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 27433 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:28:52.168 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:52.151+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:52.169 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:52.151+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:52.169 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:52.151+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:52.169 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:52.152+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:52.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:52.397+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:52.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:52.397+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:52.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:52.397+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:52.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:52.398+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:52.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:52.163+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:52.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:52.163+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:52.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:52.163+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:52.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:52.164+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:53.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:53.177+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:53.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:53.177+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:53.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:53.177+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:53.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:53.178+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:53.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:53.119+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:53.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:53.119+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:53.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:53.119+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:53.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:53.120+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:53.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:53.367+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:53.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:53.367+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:53.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:53.367+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:53.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:53.367+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:54.362 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:54.102+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:54.363 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:54.102+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:54.363 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:54.103+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:54.363 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:54.104+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:54.363 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:54.185+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:54.363 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:54.185+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:54.364 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:54.185+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:54.364 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:54.186+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:54.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:54.358+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:54.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:54.358+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:54.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:54.358+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:54.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:54.358+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:55.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:55.138+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:55.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:55.138+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:55.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:55.138+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:55.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:55.138+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:55.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:55.361+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:55.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:55.361+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:55.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:55.361+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:55.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:55.361+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:55.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:55.144+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:55.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:55.144+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:55.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:55.144+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:55.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:55.144+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:56.373 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:56.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:56.374 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:56.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:56.374 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:56.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:56.374 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:56.170+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:56.374 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:56.113+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:56.374 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:56.113+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:56.375 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:56.113+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:56.375 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:56.114+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:56.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:56.369+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:56.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:56.369+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:56.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:56.369+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:56.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:56.369+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:57.200 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:28:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:28:56.908+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 27515 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:28:57.201 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:57.150+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:57.201 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:57.150+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:57.201 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:57.150+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:57.201 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:57.152+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:57.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:57.409+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:57.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:57.409+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:57.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:57.409+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:57.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:57.410+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:57.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:57.197+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:57.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:57.197+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:57.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:57.197+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:57.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:57.198+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:58.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:58.139+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:58.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:58.139+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:58.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:58.139+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:58.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:58.140+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:58.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:58.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:58.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:58.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:58.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:58.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:58.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:58.171+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:58.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:58.409+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:58.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:58.409+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:58.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:58.409+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:58.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:58.410+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:59.454 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:59.183+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:59.454 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:59.183+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:59.454 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:59.183+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:28:59.454 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:28:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:28:59.184+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:28:59.455 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:59.184+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:59.455 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:59.184+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:59.455 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:59.184+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:28:59.455 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:28:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:28:59.185+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:28:59.770 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:59.450+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:59.770 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:59.450+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:59.770 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:59.450+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:28:59.771 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:28:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:28:59.451+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:00.386 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:00.162+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:00.387 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:00.162+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:00.387 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:00.162+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:00.387 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:00.164+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:00.387 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:00.222+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:00.387 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:00.222+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:00.388 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:00.222+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:00.388 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:00.223+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:00.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:00.403+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:00.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:00.403+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:00.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:00.403+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:00.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:00.404+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:01.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:01.369+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:01.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:01.369+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:01.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:01.369+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:01.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:01.370+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:01.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:01.206+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:01.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:01.206+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:01.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:01.206+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:01.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:01.207+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:01.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:01.203+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:01.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:01.203+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:01.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:01.203+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:01.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:01.204+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:02.040 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:29:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:29:01.912+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 27600 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:29:02.382 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:02.213+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:02.382 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:02.213+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:02.383 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:02.213+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:02.383 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:02.214+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:02.383 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:02.248+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:02.384 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:02.248+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:02.384 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:02.248+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:02.384 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:02.249+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:02.650 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:02.378+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:02.650 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:02.378+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:02.650 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:02.378+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:02.651 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:02.379+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:03.498 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:03.373+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:03.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:03.373+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:03.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:03.373+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:03.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:03.373+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:03.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:03.237+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:03.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:03.237+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:03.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:03.237+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:03.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:03.237+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:03.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:03.210+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:03.500 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:03.210+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:03.501 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:03.210+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:03.501 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:03.211+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:04.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:04.198+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:04.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:04.198+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:04.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:04.198+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:04.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:04.199+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:04.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:04.387+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:04.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:04.387+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:04.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:04.387+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:04.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:04.388+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:04.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:04.260+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:04.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:04.260+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:04.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:04.260+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:04.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:04.261+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:05.436 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:05.156+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:05.437 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:05.156+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:05.437 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:05.156+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:05.437 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:05.156+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:05.437 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:05.303+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:05.437 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:05.303+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:05.438 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:05.303+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:05.438 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:05.304+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:05.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:05.433+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:05.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:05.433+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:05.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:05.433+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:05.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:05.434+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:06.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:06.280+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:06.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:06.280+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:06.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:06.280+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:06.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:06.280+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:06.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:06.128+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:06.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:06.128+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:06.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:06.128+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:06.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:06.129+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:06.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:06.454+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:06.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:06.454+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:06.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:06.454+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:06.908 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:06.455+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:07.328 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:29:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:29:06.915+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 27681 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:29:07.329 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:07.080+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:07.329 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:07.080+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:07.329 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:07.080+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:07.329 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:07.081+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:07.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:07.444+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:07.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:07.444+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:07.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:07.444+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:07.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:07.445+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:07.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:07.325+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:07.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:07.325+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:07.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:07.325+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:07.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:07.326+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:08.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:08.116+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:08.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:08.116+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:08.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:08.116+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:08.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:08.117+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:08.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:08.297+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:08.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:08.297+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:08.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:08.297+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:08.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:08.298+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:08.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:08.405+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:08.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:08.405+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:08.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:08.405+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:08.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:08.406+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:09.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:09.148+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:09.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:09.148+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:09.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:09.148+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:09.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:09.149+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:09.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:09.286+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:09.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:09.286+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:09.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:09.286+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:09.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:09.287+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:09.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:09.438+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:09.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:09.438+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:09.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:09.438+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:09.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:09.439+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:10.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:10.116+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:10.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:10.116+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:10.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:10.116+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:10.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:10.117+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:10.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:10.305+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:10.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:10.305+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:10.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:10.305+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:10.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:10.307+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:10.768 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:10.449+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:10.768 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:10.449+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:10.768 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:10.449+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:10.768 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:10.450+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:11.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:11.092+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:11.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:11.092+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:11.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:11.092+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:11.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:11.092+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:11.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:11.260+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:11.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:11.260+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:11.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:11.260+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:11.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:11.261+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:11.747 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:11.468+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:11.747 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:11.468+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:11.748 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:11.468+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:11.748 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:11.468+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:12.044 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:29:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:29:11.918+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 27766 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:29:12.398 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:12.271+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:12.398 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:12.271+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:12.398 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:12.271+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:12.398 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:12.272+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:12.398 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:12.093+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:12.399 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:12.093+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:12.399 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:12.093+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:12.399 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:12.094+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:12.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:12.478+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:12.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:12.478+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:12.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:12.478+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:12.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:12.478+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:13.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:13.059+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:13.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:13.059+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:13.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:13.059+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:13.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:13.060+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:13.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:13.230+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:13.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:13.230+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:13.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:13.230+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:13.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:13.231+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:13.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:13.443+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:13.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:13.443+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:13.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:13.443+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:13.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:13.443+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:14.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:14.207+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:14.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:14.207+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:14.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:14.207+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:14.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:14.208+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:14.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:14.048+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:14.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:14.048+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:14.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:14.048+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:14.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:14.049+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:14.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:14.469+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:14.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:14.469+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:14.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:14.469+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:14.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:14.470+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:15.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:15.030+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:15.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:15.030+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:15.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:15.030+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:15.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:15.031+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:15.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:15.250+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:15.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:15.250+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:15.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:15.250+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:15.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:15.250+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:15.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:15.456+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:15.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:15.456+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:15.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:15.456+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:15.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:15.457+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:16.282 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:16.000+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:16.283 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:16.000+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:16.283 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:16.000+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:16.283 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:16.001+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:16.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:16.279+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:16.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:16.279+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:16.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:16.279+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:16.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:16.280+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:16.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:16.446+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:16.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:16.446+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:16.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:16.446+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:16.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:16.447+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:17.275 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:29:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:29:16.921+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 27848 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:29:17.275 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:16.976+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:17.276 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:16.976+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:17.276 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:16.976+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:17.276 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:16.977+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:17.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:17.422+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:17.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:17.422+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:17.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:17.422+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:17.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:17.424+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:17.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:17.271+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:17.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:17.271+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:17.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:17.271+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:17.660 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:17.272+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:18.286 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:18.022+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:18.286 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:18.022+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:18.286 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:18.022+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:18.287 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:18.023+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:18.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:18.282+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:18.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:18.282+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:18.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:18.282+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:18.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:18.283+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:18.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:18.389+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:18.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:18.389+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:18.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:18.389+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:18.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:18.389+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:19.296 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:18.982+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:19.296 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:18.982+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:19.297 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:18.982+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:19.297 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:18.982+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:19.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:19.341+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:19.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:19.341+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:19.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:19.341+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:19.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:19.342+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:19.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:19.292+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:19.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:19.292+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:19.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:19.292+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:19.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:19.293+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:20.311 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:20.260+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:20.312 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:20.260+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:20.312 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:20.260+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:20.312 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:20.262+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:20.312 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:20.016+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:20.313 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:20.016+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:20.313 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:20.016+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:20.313 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:20.017+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:20.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:20.308+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:20.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:20.308+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:20.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:20.308+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:20.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:20.309+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:21.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:21.055+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:21.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:21.055+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:21.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:21.055+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:21.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:21.056+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:21.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:21.277+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:21.411 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:21.277+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:21.411 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:21.277+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:21.411 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:21.278+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:21.411 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:21.279+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:21.412 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:21.279+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:21.412 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:21.279+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:21.412 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:21.280+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:22.243 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:29:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:29:21.924+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 27933 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:29:22.244 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:22.090+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:22.244 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:22.090+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:22.244 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:22.090+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:22.244 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:22.090+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:22.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:22.240+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:22.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:22.240+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:22.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:22.240+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:22.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:22.241+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:22.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:22.290+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:22.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:22.290+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:22.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:22.290+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:22.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:22.291+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:23.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:23.079+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:23.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:23.079+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:23.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:23.079+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:23.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:23.080+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:23.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:23.194+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:23.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:23.194+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:23.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:23.194+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:23.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:23.194+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:23.410 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:23.300+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:23.410 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:23.300+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:23.410 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:23.300+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:23.410 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:23.301+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:24.350 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:24.228+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:24.350 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:24.228+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:24.350 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:24.228+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:24.350 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:24.228+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:24.351 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:24.077+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:24.351 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:24.077+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:24.351 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:24.077+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:24.351 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:24.078+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:24.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:24.346+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:24.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:24.346+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:24.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:24.346+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:24.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:24.347+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:25.363 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:25.253+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:25.363 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:25.253+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:25.364 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:25.253+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:25.364 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:25.254+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:25.364 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:25.105+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:25.364 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:25.105+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:25.364 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:25.105+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:25.364 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:25.106+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:25.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:25.359+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:25.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:25.359+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:25.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:25.359+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:25.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:25.360+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:26.365 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:26.295+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:26.366 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:26.295+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:26.366 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:26.295+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:26.366 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:26.296+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:26.366 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:26.086+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:26.366 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:26.086+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:26.367 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:26.086+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:26.367 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:26.087+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:26.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:26.361+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:26.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:26.361+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:26.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:26.361+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:26.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:26.362+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:27.318 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:29:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:29:26.927+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 28015 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:29:27.319 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:27.116+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:27.319 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:27.116+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:27.319 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:27.116+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:27.319 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:27.117+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:27.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:27.315+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:27.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:27.315+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:27.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:27.315+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:27.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:27.315+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:27.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:27.401+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:27.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:27.401+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:27.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:27.401+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:27.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:27.402+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:28.396 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:28.306+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:28.396 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:28.306+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:28.397 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:28.306+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:28.397 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:28.307+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:28.397 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:28.091+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:28.397 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:28.091+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:28.397 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:28.091+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:28.397 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:28.092+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:28.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:28.392+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:28.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:28.392+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:28.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:28.392+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:28.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:28.393+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:29.383 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:29.273+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:29.383 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:29.273+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:29.383 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:29.273+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:29.384 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:29.274+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:29.384 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:29.112+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:29.384 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:29.112+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:29.384 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:29.112+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:29.384 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:29.113+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:29.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:29.379+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:29.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:29.379+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:29.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:29.379+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:29.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:29.380+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:30.395 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:30.391+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:30.395 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:30.391+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:30.396 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:30.391+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:30.396 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:30.312+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:30.396 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:30.312+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:30.396 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:30.312+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:30.396 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:30.313+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:30.397 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:30.141+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:30.397 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:30.141+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:30.397 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:30.141+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:30.397 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:30.142+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:30.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:30.392+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:31.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:31.354+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:31.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:31.354+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:31.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:31.354+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:31.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:31.354+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:31.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:31.138+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:31.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:31.138+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:31.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:31.138+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:31.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:31.139+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:31.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:31.288+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:31.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:31.288+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:31.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:31.288+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:31.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:31.289+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:32.057 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:29:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:29:31.930+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 28100 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:29:32.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:32.134+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:32.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:32.134+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:32.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:32.134+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:32.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:32.135+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:32.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:32.248+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:32.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:32.248+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:32.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:32.248+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:32.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:32.250+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:32.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:32.404+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:32.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:32.404+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:32.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:32.404+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:32.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:32.405+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:33.396 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:33.300+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:33.396 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:33.300+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:33.396 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:33.300+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:33.396 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:33.301+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:33.397 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:33.101+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:33.397 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:33.101+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:33.397 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:33.101+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:33.397 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:33.102+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:33.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:33.392+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:33.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:33.392+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:33.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:33.392+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:33.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:33.393+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:34.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:34.083+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:34.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:34.083+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:34.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:34.083+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:34.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:34.083+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:34.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:34.430+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:34.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:34.430+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:34.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:34.430+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:34.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:34.431+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:34.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:34.259+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:34.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:34.259+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:34.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:34.259+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:34.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:34.259+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:35.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:35.289+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:35.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:35.289+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:35.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:35.289+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:35.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:35.290+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:35.073+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:35.073+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:35.073+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:35.074+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:35.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:35.418+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:35.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:35.418+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:35.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:35.418+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:35.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:35.419+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:36.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:36.090+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:36.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:36.090+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:36.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:36.090+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:36.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:36.091+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:36.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:36.295+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:36.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:36.295+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:36.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:36.295+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:36.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:36.296+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:36.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:36.466+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:36.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:36.466+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:36.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:36.466+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:36.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:36.467+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:37.321 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:29:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:29:36.933+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 28182 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:29:37.322 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:37.125+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:37.322 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:37.125+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:37.322 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:37.125+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:37.323 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:37.126+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:37.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:37.317+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:37.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:37.317+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:37.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:37.317+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:37.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:37.318+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:37.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:37.506+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:37.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:37.506+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:37.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:37.506+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:37.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:37.507+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:38.478 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:38.342+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:38.479 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:38.342+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:38.479 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:38.342+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:38.479 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:38.343+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:38.479 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:38.158+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:38.479 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:38.158+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:38.480 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:38.158+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:38.480 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:38.159+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:38.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:38.475+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:38.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:38.475+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:38.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:38.475+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:38.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:38.475+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:39.465 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:39.320+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:39.465 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:39.320+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:39.465 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:39.320+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:39.466 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:39.321+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:39.466 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:39.186+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:39.466 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:39.186+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:39.466 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:39.186+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:39.466 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:39.187+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:39.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:39.461+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:39.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:39.461+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:39.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:39.461+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:39.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:39.463+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:40.446 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:40.281+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:40.446 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:40.281+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:40.446 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:40.281+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:40.446 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:40.282+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:40.446 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:40.189+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:40.446 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:40.189+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:40.447 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:40.189+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:40.447 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:40.190+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:40.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:40.442+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:40.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:40.442+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:40.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:40.442+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:40.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:40.443+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:41.472 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:41.189+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:41.472 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:41.189+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:41.472 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:41.189+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:41.472 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:41.190+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:41.473 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:41.299+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:41.473 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:41.299+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:41.473 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:41.299+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:41.473 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:41.300+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:41.749 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:41.468+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:41.749 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:41.468+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:41.749 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:41.468+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:41.750 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:41.469+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:42.062 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:29:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:29:41.936+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 28267 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:29:42.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:42.180+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:42.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:42.180+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:42.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:42.180+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:42.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:42.182+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:42.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:42.268+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:42.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:42.268+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:42.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:42.268+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:42.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:42.269+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:42.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:42.428+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:42.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:42.428+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:42.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:42.428+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:42.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:42.429+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:43.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:43.271+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:43.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:43.271+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:43.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:43.271+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:43.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:43.272+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:43.176+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:43.176+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:43.176+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:43.177+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:43.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:43.444+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:43.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:43.444+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:43.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:43.444+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:43.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:43.445+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:44.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:44.406+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:44.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:44.406+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:44.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:44.406+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:44.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:44.407+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:44.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:44.303+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:44.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:44.303+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:44.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:44.303+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:44.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:44.304+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:44.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:44.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:44.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:44.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:44.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:44.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:44.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:44.171+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:45.405 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:45.139+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:45.405 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:45.139+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:45.406 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:45.139+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:45.406 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:45.140+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:45.406 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:45.312+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:45.406 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:45.312+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:45.406 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:45.312+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:45.406 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:45.314+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:45.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:45.401+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:45.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:45.401+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:45.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:45.401+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:45.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:45.402+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:46.361 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:46.336+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:46.362 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:46.336+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:46.362 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:46.336+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:46.362 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:46.337+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:46.362 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:46.097+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:46.362 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:46.097+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:46.362 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:46.097+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:46.363 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:46.098+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:46.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:46.357+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:46.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:46.357+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:46.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:46.357+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:46.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:46.358+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:47.333 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:29:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:29:46.939+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 28349 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:29:47.334 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:47.079+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:47.334 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:47.079+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:47.334 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:47.079+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:47.334 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:47.081+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:47.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:47.330+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:47.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:47.330+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:47.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:47.330+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:47.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:47.330+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:47.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:47.341+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:47.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:47.341+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:47.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:47.341+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:47.659 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:47.342+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:48.382 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:48.094+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:48.382 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:48.094+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:48.382 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:48.094+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:48.382 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:48.095+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:48.383 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:48.312+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:48.383 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:48.312+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:48.383 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:48.312+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:48.383 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:48.312+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:48.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:48.378+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:48.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:48.378+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:48.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:48.378+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:48.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:48.378+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:49.354 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:49.099+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:49.355 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:49.099+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:49.355 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:49.099+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:49.355 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:49.100+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:49.355 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:49.290+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:49.355 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:49.290+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:49.355 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:49.290+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:49.356 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:49.291+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:49.660 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:49.351+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:49.660 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:49.351+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:49.660 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:49.351+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:49.660 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:49.352+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:50.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:50.111+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:50.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:50.111+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:50.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:50.111+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:50.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:50.112+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:50.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:50.332+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:50.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:50.332+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:50.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:50.332+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:50.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:50.332+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:50.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:50.314+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:50.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:50.314+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:50.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:50.314+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:50.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:50.315+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:51.384 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:51.121+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:51.384 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:51.121+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:51.384 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:51.121+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:51.384 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:51.122+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:51.385 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:51.270+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:51.385 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:51.270+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:51.385 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:51.270+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:51.385 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:51.271+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:51.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:51.380+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:51.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:51.380+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:51.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:51.380+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:51.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:51.381+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:52.074 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:29:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:29:51.942+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 28433 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:29:52.334 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:52.257+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:52.334 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:52.257+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:52.334 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:52.257+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:52.334 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:52.257+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:52.334 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:52.080+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:52.335 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:52.080+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:52.335 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:52.080+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:52.335 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:52.081+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:52.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:52.372+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:52.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:52.372+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:52.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:52.372+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:52.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:52.373+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:53.329 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:53.052+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:53.329 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:53.052+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:53.329 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:53.052+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:53.329 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:53.053+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:53.329 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:53.291+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:53.330 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:53.291+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:53.330 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:53.291+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:53.330 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:53.292+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:53.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:53.325+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:53.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:53.325+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:53.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:53.325+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:53.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:53.326+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:54.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:54.277+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:54.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:54.277+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:54.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:54.277+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:54.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:54.277+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:54.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:54.297+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:54.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:54.297+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:54.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:54.297+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:54.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:54.298+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:54.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:54.056+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:54.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:54.056+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:54.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:54.056+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:54.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:54.057+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:55.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:55.097+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:55.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:55.097+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:55.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:55.097+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:55.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:55.098+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:55.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:55.289+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:55.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:55.289+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:55.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:55.289+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:55.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:55.290+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:55.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:55.243+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:55.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:55.243+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:55.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:55.243+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:24.127047+0000 (oldest deadline 2024-02-09T22:19:44.127047+0000) 2024-02-09T22:29:55.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:55.243+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:56.324 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:56.248+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:56.324 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:56.051+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:56.324 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:56.051+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:56.324 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:56.051+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:56.325 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:56.052+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:56.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:56.320+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:56.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:56.320+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:56.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:56.320+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:56.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:56.321+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:57.255 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:29:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:29:56.945+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 28515 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:29:57.256 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:57.029+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:57.256 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:57.029+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:57.256 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:57.029+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:57.256 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:57.029+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:57.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:57.250+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:57.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:57.327+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:57.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:57.327+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:57.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:57.327+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:57.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:57.329+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:58.324 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:58.210+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:58.324 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:58.063+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:58.325 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:58.063+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:58.325 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:58.063+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:58.325 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:58.064+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:58.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:58.320+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:58.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:58.320+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:58.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:58.320+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:58.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:58.321+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:59.328 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:29:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:29:59.241+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:29:59.329 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:59.071+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:59.329 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:59.071+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:59.329 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:59.071+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:29:59.329 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:29:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:29:59.072+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:29:59.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:59.325+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:59.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:59.325+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:59.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:59.325+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:29:59.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:29:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:29:59.326+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:00.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:00.252+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:00.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:00.084+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:00.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:00.084+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:00.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:00.084+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:00.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:00.085+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:00.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:00.285+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:30:00.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:00.285+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:30:00.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:00.285+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:30:00.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:00.286+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:01.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:01.261+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:01.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:01.327+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:30:01.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:01.327+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:30:01.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:01.327+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:30:01.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:01.328+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:01.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:01.133+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:01.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:01.133+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:01.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:01.133+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:01.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:01.133+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:02.244 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:30:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:30:01.948+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 28599 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:30:02.245 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:02.159+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:02.245 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:02.159+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:02.245 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:02.159+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:02.245 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:02.160+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:02.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:02.241+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:02.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:02.305+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:30:02.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:02.305+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:30:02.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:02.305+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:30:02.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:02.306+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:03.498 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:03.283+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:03.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:03.330+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:30:03.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:03.330+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:30:03.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:03.330+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:30.079358+0000 (oldest deadline 2024-02-09T22:19:50.079358+0000) 2024-02-09T22:30:03.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:03.331+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:03.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:03.163+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:03.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:03.163+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:03.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:03.163+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:03.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:03.164+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:04.621 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:04.366+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:04.622 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:04.280+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:04.622 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:04.197+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:04.622 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:04.197+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:04.623 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:04.197+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:04.623 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:04.198+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:05.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:05.403+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:05.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:05.330+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:05.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:05.165+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:05.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:05.165+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:05.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:05.165+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:05.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:05.166+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:06.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:06.371+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:06.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:06.356+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:06.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:06.142+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:06.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:06.142+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:06.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:06.142+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:06.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:06.143+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:07.327 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:30:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:30:06.951+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 28681 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:30:07.327 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:07.189+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:07.327 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:07.189+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:07.327 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:07.189+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:07.328 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:07.190+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:07.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:07.407+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:07.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:07.323+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:08.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:08.418+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:08.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:08.369+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:08.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:08.238+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:08.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:08.238+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:08.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:08.238+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:08.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:08.239+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:09.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:09.460+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:09.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:09.376+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:09.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:09.276+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:09.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:09.276+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:09.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:09.276+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:09.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:09.277+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:10.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:10.447+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:10.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:10.418+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:10.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:10.247+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:10.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:10.247+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:10.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:10.247+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:10.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:10.248+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:11.459 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:11.402+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:11.460 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:11.205+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:11.460 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:11.205+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:11.460 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:11.205+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:11.460 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:11.206+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:11.751 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:11.456+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:12.084 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:30:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:30:11.954+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 28766 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:30:12.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:12.218+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:12.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:12.218+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:12.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:12.218+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:12.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:12.219+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:12.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:12.466+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:12.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:12.422+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:13.462 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:13.195+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:13.462 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:13.195+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:13.462 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:13.195+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:13.462 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:13.196+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:13.839 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:13.458+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:13.839 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:13.472+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:14.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:14.470+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:14.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:14.470+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:14.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:14.240+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:14.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:14.240+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:14.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:14.240+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:14.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:14.241+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:15.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:15.422+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:15.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:15.476+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:15.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:15.243+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:15.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:15.243+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:15.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:15.243+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:15.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:15.244+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:16.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:16.414+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:16.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:16.445+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:16.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:16.231+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:16.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:16.231+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:16.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:16.231+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:16.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:16.231+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:17.220 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:30:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:30:16.958+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 28847 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:30:17.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:17.373+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:17.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:17.415+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:17.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:17.216+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:17.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:17.216+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:17.660 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:17.216+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:17.660 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:17.217+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:18.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:18.397+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:18.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:18.198+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:18.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:18.198+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:18.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:18.198+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:19:40.763702+0000 (oldest deadline 2024-02-09T22:20:00.763702+0000) 2024-02-09T22:30:18.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:18.199+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:18.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:18.327+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:18.660 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:18.327+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:18.660 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:18.327+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:18.660 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:18.327+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:19.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:19.405+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:19.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:19.208+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:19.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:19.292+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:19.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:19.292+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:19.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:19.292+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:19.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:19.293+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:20.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:20.416+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:20.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:20.177+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:20.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:20.255+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:20.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:20.255+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:20.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:20.255+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:20.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:20.255+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:21.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:21.412+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:21.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:21.181+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:21.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:21.268+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:21.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:21.268+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:21.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:21.268+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:21.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:21.268+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:22.090 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:30:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:30:21.961+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 28932 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:30:22.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:22.147+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:22.305+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:22.305+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:22.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:22.305+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:22.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:22.306+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:22.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:22.426+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:23.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:23.115+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:23.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:23.277+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:23.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:23.277+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:23.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:23.277+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:23.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:23.277+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:23.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:23.449+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:24.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:24.076+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:24.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:24.259+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:24.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:24.259+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:24.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:24.259+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:24.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:24.260+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:24.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:24.439+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:25.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:25.073+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:25.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:25.246+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:25.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:25.246+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:25.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:25.246+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:25.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:25.247+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:25.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:25.476+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:26.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:26.112+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:26.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:26.230+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:26.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:26.230+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:26.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:26.230+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:26.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:26.231+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:26.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:26.443+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:27.219 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:30:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:30:26.964+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 29014 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:30:27.219 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:27.077+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:27.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:27.406+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:27.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:27.215+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:27.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:27.215+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:27.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:27.215+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:27.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:27.216+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:28.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:28.068+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:28.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:28.203+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:28.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:28.203+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:28.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:28.203+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:28.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:28.204+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:28.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:28.428+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:29.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:29.104+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:29.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:29.178+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:29.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:29.178+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:29.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:29.178+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:29.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:29.179+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:29.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:29.416+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:29.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:29.416+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:29.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:29.416+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:29.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:29.417+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:30.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:30.091+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:30.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:30.177+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:30.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:30.177+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:30.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:30.177+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:30.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:30.178+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:30.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:30.421+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:30.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:30.421+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:30.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:30.421+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:30.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:30.422+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:31.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:31.110+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:31.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:31.188+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:31.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:31.188+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:31.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:31.188+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:31.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:31.189+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:31.753 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:31.417+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:31.753 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:31.417+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:31.753 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:31.417+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:31.753 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:31.418+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:32.091 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:30:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:30:31.966+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 29099 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:30:32.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:32.086+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:32.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:32.218+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:32.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:32.218+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:32.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:32.218+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:32.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:32.218+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:32.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:32.409+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:32.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:32.409+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:32.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:32.409+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:32.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:32.410+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:33.373 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:33.063+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:33.374 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:33.215+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:33.374 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:33.215+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:33.374 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:33.215+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:33.374 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:33.216+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:33.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:33.369+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:33.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:33.369+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:33.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:33.369+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:33.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:33.370+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:34.390 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:34.172+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:34.390 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:34.172+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:34.390 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:34.172+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:34.391 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:34.173+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:34.391 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:34.035+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:34.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:34.386+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:34.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:34.386+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:34.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:34.386+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:34.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:34.386+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:35.390 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:34.996+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:35.390 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:35.187+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:35.391 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:35.187+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:35.391 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:35.187+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:35.391 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:35.187+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:35.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:35.386+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:35.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:35.386+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:35.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:35.386+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:35.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:35.388+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:36.395 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:36.181+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:36.396 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:36.181+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:36.396 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:36.181+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:36.396 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:36.181+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:36.396 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:36.003+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:36.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:36.392+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:36.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:36.392+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:36.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:36.392+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:36.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:36.393+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:37.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:37.028+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:37.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:30:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:30:36.970+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 29181 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:30:37.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:37.138+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:37.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:37.138+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:37.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:37.138+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:37.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:37.139+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:37.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:37.438+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:37.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:37.438+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:37.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:37.438+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:37.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:37.439+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:38.397 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:38.077+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:38.398 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:38.140+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:38.398 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:38.140+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:38.398 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:38.140+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:38.398 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:38.141+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:38.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:38.393+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:38.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:38.393+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:38.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:38.393+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:38.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:38.395+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:39.353 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:39.067+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:39.353 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:39.159+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:39.353 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:39.159+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:39.353 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:39.159+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:39.354 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:39.160+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:39.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:39.349+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:39.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:39.349+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:39.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:39.349+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:39.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:39.350+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:40.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:40.075+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:40.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:40.075+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:40.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:40.075+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:40.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:40.076+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:40.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:40.312+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:40.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:40.312+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:40.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:40.312+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:40.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:40.313+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:40.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:40.127+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:40.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:40.127+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:40.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:40.127+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:40.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:40.128+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:41.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:41.279+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:41.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:41.279+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:41.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:41.279+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:41.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:41.280+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:41.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:41.087+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:41.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:41.087+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:41.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:41.087+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:41.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:41.088+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:41.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:41.046+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:41.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:41.046+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:41.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:41.046+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:41.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:41.047+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:42.063 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:30:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:30:41.974+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 29265 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:30:42.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:42.262+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:42.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:42.262+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:42.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:42.262+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:42.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:42.262+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:42.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:42.133+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:42.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:42.133+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:42.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:42.133+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:42.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:42.134+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:42.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:42.059+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:42.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:42.059+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:42.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:42.059+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:42.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:42.060+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:43.035+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:43.035+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:43.035+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:43.036+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:43.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:43.243+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:43.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:43.243+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:43.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:43.243+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:43.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:43.244+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:43.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:43.105+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:43.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:43.105+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:43.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:43.105+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:43.410 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:43.106+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:44.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:44.205+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:44.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:44.205+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:44.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:44.205+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:44.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:44.205+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:44.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:44.120+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:44.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:44.120+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:44.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:44.120+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:44.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:44.121+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:44.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:44.083+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:44.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:44.083+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:44.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:44.083+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:44.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:44.084+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:45.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:45.175+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:45.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:45.175+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:45.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:45.175+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:45.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:45.176+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:45.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:45.151+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:45.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:45.151+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:45.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:45.151+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:45.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:45.152+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:45.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:45.046+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:45.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:45.046+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:45.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:45.046+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:45.411 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:45.047+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:46.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:46.171+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:46.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:46.171+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:46.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:46.171+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:46.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:46.172+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:46.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:46.121+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:46.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:46.122+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:46.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:46.122+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:46.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:46.122+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:46.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:46.088+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:46.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:46.088+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:46.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:46.088+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:46.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:46.089+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:47.308 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:47.215+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:47.309 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:47.215+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:47.309 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:47.215+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:47.309 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:47.216+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:47.309 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:30:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:30:46.977+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 29347 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:30:47.310 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:47.168+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:47.310 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:47.168+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:47.310 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:47.168+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:47.310 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:47.170+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:47.310 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:47.104+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:47.310 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:47.104+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:47.311 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:47.104+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:47.311 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:47.105+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:48.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:48.191+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:48.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:48.191+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:48.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:48.191+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:48.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:48.192+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:48.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:48.117+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:48.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:48.117+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:48.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:48.117+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:48.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:48.118+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:48.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:48.149+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:48.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:48.149+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:48.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:48.149+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:48.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:48.150+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:49.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:49.234+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:49.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:49.234+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:49.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:49.234+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:49.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:49.235+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:49.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:49.103+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:49.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:49.103+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:49.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:49.103+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:49.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:49.104+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:49.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:49.079+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:49.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:49.079+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:49.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:49.079+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:49.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:49.080+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:50.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:50.139+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:50.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:50.139+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:50.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:50.139+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:50.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:50.140+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:50.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:50.194+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:50.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:50.194+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:50.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:50.194+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:50.410 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:50.195+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:50.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:50.050+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:50.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:50.050+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:50.411 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:50.050+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:50.411 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:50.051+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:51.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:51.163+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:51.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:51.163+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:51.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:51.163+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:51.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:51.164+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:51.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:51.091+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:51.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:51.091+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:51.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:51.091+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:51.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:51.093+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:51.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:51.035+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:51.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:51.035+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:51.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:51.035+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:51.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:51.037+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:52.008 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:30:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:30:51.980+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 29431 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:30:52.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:52.125+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:52.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:52.125+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:52.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:52.125+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:52.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:52.126+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:52.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:52.050+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:52.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:52.050+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:52.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:52.050+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:52.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:52.050+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:52.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:52.055+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:52.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:52.055+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:52.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:52.055+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:52.410 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:52.056+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:53.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:53.103+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:53.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:53.103+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:53.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:53.103+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:53.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:53.104+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:53.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:53.069+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:53.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:53.069+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:53.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:53.069+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:53.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:53.070+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:53.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:53.037+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:53.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:53.037+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:53.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:53.037+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:53.410 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:53.038+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:54.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:54.085+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:54.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:54.085+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:54.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:54.085+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:54.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:54.086+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:54.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:54.144+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:54.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:54.144+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:54.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:54.144+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:54.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:54.144+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:54.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:54.088+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:54.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:54.088+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:54.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:54.088+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:54.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:54.089+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:55.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:55.112+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:55.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:55.112+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:55.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:55.112+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:55.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:55.113+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:55.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:55.177+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:55.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:55.177+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:55.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:55.177+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:55.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:55.178+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:55.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:55.114+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:55.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:55.114+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:55.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:55.114+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:55.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:55.115+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:56.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:56.136+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:56.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:56.136+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:56.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:56.136+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:56.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:56.137+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:56.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:56.157+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:56.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:56.157+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:56.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:56.157+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:56.410 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:56.157+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:56.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:56.066+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:56.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:56.066+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:56.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:56.066+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:56.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:56.067+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:57.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:30:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:30:56.984+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 29513 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:30:57.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:57.160+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:57.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:57.160+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:57.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:57.160+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:57.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:57.161+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:57.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:57.206+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:57.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:57.206+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:57.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:57.206+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:57.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:57.207+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:57.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:57.113+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:57.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:57.113+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:57.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:57.113+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:57.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:57.114+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:58.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:58.123+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:58.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:58.123+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:58.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:58.123+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:58.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:58.123+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:58.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:58.178+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:58.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:58.178+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:58.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:58.178+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:58.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:58.179+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:58.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:58.196+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:58.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:58.196+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:58.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:58.196+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:58.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:58.197+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:59.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:59.171+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:59.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:59.171+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:59.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:59.171+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:30:59.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:30:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:30:59.172+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:30:59.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:59.213+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:59.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:59.213+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:59.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:59.213+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:30:59.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:30:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:30:59.213+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:30:59.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:59.089+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:59.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:59.089+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:59.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:59.089+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:30:59.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:30:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:30:59.090+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:00.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:00.127+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:00.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:00.127+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:00.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:00.127+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:00.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:00.128+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:00.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:00.087+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:00.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:00.087+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:00.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:00.087+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:00.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:00.088+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:00.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:00.173+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:00.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:00.173+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:00.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:00.173+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:00.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:00.173+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:01.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:01.222+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:01.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:01.222+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:01.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:01.222+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:01.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:01.223+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:01.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:01.122+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:01.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:01.122+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:01.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:01.122+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:01.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:01.123+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:01.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:01.040+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:01.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:01.040+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:01.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:01.040+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:01.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:01.041+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:02.014 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:31:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:31:01.987+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 29597 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:31:02.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:02.196+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:02.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:02.196+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:02.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:02.196+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:02.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:02.196+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:02.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:02.102+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:02.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:02.102+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:02.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:02.102+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:02.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:02.102+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:02.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:02.067+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:02.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:02.067+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:02.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:02.067+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:02.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:02.068+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:03.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:03.092+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:03.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:03.092+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:03.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:03.092+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:03.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:03.093+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:03.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:03.110+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:03.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:03.110+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:03.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:03.110+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:03.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:03.111+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:03.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:03.211+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:03.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:03.211+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:03.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:03.211+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:03.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:03.212+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:04.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:04.214+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:04.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:04.214+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:04.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:04.214+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:04.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:04.214+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:04.131+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:04.131+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:04.131+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:04.132+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:04.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:04.084+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:04.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:04.084+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:04.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:04.084+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:04.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:04.085+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:05.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:05.081+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:05.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:05.081+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:05.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:05.081+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:05.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:05.082+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:05.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:05.219+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:05.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:05.219+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:05.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:05.219+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:05.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:05.220+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:05.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:05.144+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:05.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:05.144+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:05.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:05.144+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:05.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:05.144+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:06.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:06.099+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:06.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:06.099+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:06.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:06.099+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:06.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:06.100+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:06.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:06.238+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:06.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:06.238+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:06.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:06.238+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:06.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:06.240+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:06.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:06.106+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:06.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:06.106+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:06.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:06.106+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:06.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:06.107+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:07.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:31:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:31:06.990+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 29680 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:31:07.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:07.080+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:07.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:07.081+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:07.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:07.081+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:07.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:07.082+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:07.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:07.190+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:07.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:07.190+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:07.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:07.190+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:07.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:07.191+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:07.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:07.119+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:07.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:07.119+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:07.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:07.119+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:07.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:07.120+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:08.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:08.116+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:08.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:08.116+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:08.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:08.116+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:08.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:08.118+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:08.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:08.188+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:08.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:08.188+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:08.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:08.188+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:08.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:08.189+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:08.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:08.114+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:08.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:08.114+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:08.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:08.114+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:08.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:08.115+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:09.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:09.074+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:09.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:09.074+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:09.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:09.074+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:09.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:09.076+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:09.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:09.179+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:09.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:09.179+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:09.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:09.179+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:09.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:09.180+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:09.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:09.153+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:09.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:09.153+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:09.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:09.153+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:09.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:09.154+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:10.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:10.082+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:10.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:10.082+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:10.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:10.082+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:10.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:10.082+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:10.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:10.194+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:10.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:10.194+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:10.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:10.194+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:10.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:10.195+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:10.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:10.115+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:10.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:10.115+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:10.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:10.115+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:10.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:10.116+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:11.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:11.037+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:11.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:11.037+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:11.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:11.037+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:11.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:11.038+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:11.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:11.134+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:11.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:11.134+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:11.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:11.134+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:11.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:11.135+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:11.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:11.238+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:11.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:11.238+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:11.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:11.238+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:11.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:11.239+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:12.011 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:31:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:31:11.993+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 29764 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:31:12.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:12.011+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:12.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:12.011+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:12.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:12.011+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:12.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:12.012+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:12.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:12.204+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:12.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:12.204+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:12.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:12.204+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:12.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:12.205+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:12.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:12.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:12.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:12.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:12.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:12.170+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:12.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:12.171+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:13.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:12.970+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:13.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:12.970+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:13.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:12.970+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:13.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:12.970+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:13.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:13.203+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:13.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:13.203+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:13.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:13.203+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:13.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:13.204+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:13.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:13.128+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:13.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:13.128+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:13.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:13.128+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:13.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:13.129+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:14.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:13.946+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:14.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:13.946+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:14.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:13.946+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:14.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:13.947+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:14.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:14.165+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:14.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:14.165+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:14.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:14.165+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:14.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:14.166+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:14.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:14.106+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:14.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:14.106+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:14.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:14.106+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:14.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:14.107+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:15.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:14.899+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:15.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:14.899+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:15.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:14.899+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:15.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:14.900+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:15.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:15.072+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:15.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:15.072+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:15.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:15.072+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:15.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:15.073+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:15.660 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:15.216+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:15.660 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:15.216+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:15.660 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:15.216+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:15.660 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:15.217+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:16.223 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:15.915+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:16.223 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:15.915+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:16.223 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:15.915+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:16.223 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:15.916+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:16.223 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:16.045+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:16.224 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:16.045+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:16.224 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:16.045+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:16.224 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:16.046+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:16.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:16.219+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:16.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:16.219+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:16.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:16.219+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:16.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:16.220+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:17.157 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:31:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:31:16.997+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 29847 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:31:17.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:16.899+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:17.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:16.899+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:17.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:16.899+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:17.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:16.900+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:17.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:17.039+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:17.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:17.039+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:17.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:17.039+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:17.158 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:17.040+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:17.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:17.210+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:17.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:17.210+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:17.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:17.210+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:17.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:17.210+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:18.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:17.940+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:18.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:17.940+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:18.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:17.940+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:18.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:17.941+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:18.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:18.167+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:18.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:18.167+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:18.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:18.167+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:18.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:18.168+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:18.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:18.022+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:18.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:18.022+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:18.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:18.022+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:18.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:18.023+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:19.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:18.980+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:19.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:18.980+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:19.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:18.980+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:19.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:18.981+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:19.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:19.131+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:19.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:19.131+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:19.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:19.131+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:19.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:19.132+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:19.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:19.067+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:19.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:19.067+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:19.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:19.067+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:19.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:19.068+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:20.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:20.020+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:20.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:20.020+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:20.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:20.020+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:20.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:20.021+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:20.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:20.124+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:20.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:20.124+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:20.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:20.124+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:20.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:20.126+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:20.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:20.107+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:20.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:20.107+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:20.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:20.107+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:20.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:20.108+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:21.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:20.986+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:21.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:20.986+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:21.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:20.986+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:21.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:20.986+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:21.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:21.110+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:21.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:21.110+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:21.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:21.110+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:21.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:21.110+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:21.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:21.117+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:21.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:21.117+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:21.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:21.117+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:21.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:21.118+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:22.019 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:31:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:31:22.000+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 29931 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:31:22.020 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:21.998+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:22.020 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:21.998+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:22.020 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:21.998+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:22.020 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:21.999+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:22.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:22.084+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:22.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:22.084+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:22.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:22.084+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:22.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:22.085+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:22.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:22.117+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:22.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:22.117+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:22.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:22.117+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:22.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:22.118+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:23.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:23.127+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:23.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:23.127+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:23.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:23.127+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:23.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:23.127+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:23.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:23.021+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:23.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:23.021+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:23.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:23.021+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:23.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:23.022+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:23.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:23.059+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:23.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:23.059+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:23.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:23.059+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:23.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:23.060+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:24.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:24.042+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:24.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:24.042+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:24.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:24.042+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:24.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:24.043+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:24.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:24.138+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:24.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:24.138+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:24.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:24.138+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:24.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:24.139+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:24.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:24.073+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:24.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:24.073+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:24.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:24.073+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:24.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:24.074+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:25.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:25.051+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:25.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:25.051+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:25.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:25.051+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:25.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:25.052+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:25.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:25.166+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:25.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:25.166+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:25.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:25.166+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:25.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:25.168+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:25.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:25.088+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:25.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:25.088+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:25.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:25.088+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:25.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:25.089+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:26.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:26.007+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:26.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:26.007+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:26.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:26.007+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:26.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:26.008+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:26.125+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:26.125+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:26.125+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:26.126+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:26.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:26.181+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:26.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:26.181+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:26.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:26.181+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:26.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:26.182+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:27.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:31:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:31:27.003+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 30014 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:31:27.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:27.047+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:27.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:27.047+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:27.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:27.047+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:27.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:27.048+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:27.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:27.136+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:27.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:27.136+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:27.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:27.136+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:27.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:27.137+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:27.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:27.150+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:27.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:27.150+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:27.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:27.150+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:27.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:27.150+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:28.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:28.004+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:28.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:28.004+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:28.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:28.004+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:28.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:28.006+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:28.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:28.181+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:28.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:28.181+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:28.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:28.181+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:28.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:28.182+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:28.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:28.141+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:28.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:28.141+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:28.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:28.141+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:28.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:28.142+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:29.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:29.176+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:29.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:29.176+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:29.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:29.176+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:29.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:29.177+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:29.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:29.006+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:29.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:29.006+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:29.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:29.006+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:29.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:29.006+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:29.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:29.142+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:29.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:29.142+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:29.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:29.142+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:29.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:29.143+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:30.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:30.054+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:30.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:30.054+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:30.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:30.054+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:30.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:30.054+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:30.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:30.107+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:30.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:30.107+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:30.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:30.107+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:30.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:30.108+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:30.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:30.217+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:30.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:30.217+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:30.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:30.217+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:30.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:30.218+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:31.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:31.024+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:31.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:31.024+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:31.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:31.024+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:31.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:31.025+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:31.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:31.247+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:31.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:31.247+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:31.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:31.247+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:31.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:31.248+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:31.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:31.060+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:31.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:31.060+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:31.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:31.060+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:31.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:31.060+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:32.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:31:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:31:32.007+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 30098 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:31:32.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:32.009+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:32.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:32.009+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:32.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:32.009+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:32.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:32.010+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:32.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:32.201+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:32.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:32.201+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:32.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:32.201+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:32.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:32.202+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:32.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:32.054+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:32.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:32.054+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:32.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:32.054+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:32.410 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:32.054+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:33.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:33.013+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:33.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:33.013+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:33.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:33.013+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:33.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:33.014+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:33.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:33.239+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:33.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:33.239+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:33.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:33.239+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:33.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:33.240+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:33.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:33.086+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:33.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:33.086+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:33.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:33.086+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:33.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:33.087+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:34.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:34.047+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:34.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:34.047+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:34.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:34.047+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:34.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:34.048+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:34.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:34.062+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:34.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:34.062+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:34.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:34.062+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:34.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:34.063+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:34.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:34.286+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:34.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:34.286+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:34.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:34.286+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:34.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:34.287+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:35.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:35.051+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:35.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:35.051+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:35.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:35.051+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:35.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:35.051+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:35.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:35.277+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:35.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:35.277+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:35.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:35.277+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:35.409 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:35.278+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:35.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:35.102+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:35.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:35.102+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:35.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:35.102+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:35.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:35.102+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:36.325 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:36.058+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:36.325 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:36.058+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:36.326 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:36.058+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:36.326 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:36.059+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:36.326 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:36.150+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:36.326 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:36.150+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:36.326 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:36.150+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:36.326 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:36.151+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:36.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:36.321+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:36.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:36.321+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:36.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:36.321+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:36.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:36.322+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:37.334 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:31:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:31:37.010+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 30181 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:31:37.335 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:37.050+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:37.335 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:37.050+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:37.335 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:37.050+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:37.335 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:37.051+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:37.335 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:37.149+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:37.336 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:37.149+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:37.336 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:37.149+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:37.336 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:37.150+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:37.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:37.331+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:37.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:37.331+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:37.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:37.331+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:37.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:37.332+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:38.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:38.293+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:38.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:38.293+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:38.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:38.293+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:38.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:38.293+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:38.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:38.060+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:38.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:38.060+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:38.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:38.060+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:38.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:38.061+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:38.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:38.154+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:38.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:38.154+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:38.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:38.154+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:38.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:38.154+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:39.321 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:39.042+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:39.321 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:39.042+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:39.321 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:39.042+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:39.322 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:39.042+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:39.322 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:39.107+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:39.322 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:39.107+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:39.322 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:39.107+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:39.322 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:39.108+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:39.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:39.317+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:39.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:39.317+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:39.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:39.317+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:39.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:39.318+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:40.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:40.064+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:40.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:40.064+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:40.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:40.064+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:40.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:40.065+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:40.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:40.287+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:40.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:40.287+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:40.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:40.287+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:40.408 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:40.288+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:40.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:40.114+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:40.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:40.114+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:40.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:40.114+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:40.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:40.115+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:41.304 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:41.128+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:41.305 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:41.128+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:41.305 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:41.128+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:41.305 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:41.129+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:41.305 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:41.031+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:41.305 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:41.031+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:41.306 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:41.031+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:41.306 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:41.032+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:41.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:41.301+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:41.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:41.301+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:41.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:41.301+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:41.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:41.301+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:42.018 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:41.989+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:42.018 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:41.989+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:42.018 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:41.989+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:42.019 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:41.990+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:42.326 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:31:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:31:42.014+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 30265 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:31:42.327 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:42.080+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:42.327 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:42.080+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:42.327 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:42.080+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:42.327 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:42.080+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:42.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:42.323+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:42.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:42.323+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:42.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:42.323+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:42.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:42.323+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:43.306 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:42.976+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:43.306 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:42.976+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:43.306 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:42.976+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:43.306 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:42.977+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:43.307 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:43.096+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:43.307 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:43.096+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:43.307 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:43.096+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:43.307 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:43.097+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:43.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:43.302+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:43.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:43.302+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:43.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:43.302+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:43.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:43.303+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:44.322 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:43.978+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:44.323 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:43.978+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:44.323 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:43.978+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:44.323 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:43.979+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:44.323 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:44.144+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:44.323 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:44.144+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:44.324 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:44.144+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:44.324 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:44.145+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:44.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:44.319+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:44.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:44.319+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:44.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:44.319+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:44.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:44.319+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:45.328 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:45.013+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:45.328 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:45.013+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:45.329 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:45.013+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:45.329 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:45.013+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:45.329 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:45.147+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:45.329 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:45.147+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:45.329 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:45.147+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:45.329 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:45.148+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:45.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:45.324+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:45.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:45.324+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:45.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:45.324+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:45.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:45.325+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:46.354 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:46.011+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:46.354 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:46.011+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:46.355 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:46.011+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:46.355 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:46.012+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:46.355 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:46.122+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:46.355 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:46.122+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:46.355 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:46.122+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:46.355 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:46.123+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:46.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:46.350+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:46.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:46.350+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:46.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:46.350+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:46.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:46.352+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:47.312 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:31:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:31:47.017+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 30348 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:31:47.313 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:46.974+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:47.313 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:46.974+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:47.313 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:46.974+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:47.313 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:46.975+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:47.313 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:47.084+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:47.314 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:47.084+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:47.314 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:47.084+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:47.314 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:47.085+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:47.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:47.309+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:47.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:47.309+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:47.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:47.309+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:47.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:47.310+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:48.352 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:47.950+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:48.352 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:47.950+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:48.352 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:47.950+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:48.352 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:47.951+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:48.353 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:48.124+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:48.353 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:48.124+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:48.353 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:48.124+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:48.353 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:48.125+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:48.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:48.348+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:48.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:48.348+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:48.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:48.348+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:48.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:48.349+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:49.393 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:48.967+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:49.393 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:48.967+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:49.393 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:48.967+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:49.394 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:48.967+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:49.394 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:49.128+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:49.394 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:49.128+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:49.394 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:49.128+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:49.394 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:49.129+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:49.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:49.389+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:49.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:49.389+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:49.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:49.389+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:49.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:49.390+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:50.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:50.141+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:50.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:50.141+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:50.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:50.141+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:50.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:50.142+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:50.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:49.990+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:50.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:49.990+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:50.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:49.990+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:50.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:49.991+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:50.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:50.417+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:50.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:50.417+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:50.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:50.417+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:50.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:50.417+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:51.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:51.110+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:51.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:51.110+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:51.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:51.110+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:51.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:51.111+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:51.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:51.022+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:51.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:51.022+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:51.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:51.022+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:51.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:51.023+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:51.758 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:51.467+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:51.758 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:51.467+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:51.758 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:51.467+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:51.759 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:51.467+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:52.008 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:52.003+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:52.008 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:52.003+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:52.008 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:52.003+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:52.008 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:52.004+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:52.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:31:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:31:52.020+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 30432 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:31:52.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:52.125+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:52.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:52.125+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:52.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:52.125+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:52.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:52.125+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:52.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:52.431+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:52.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:52.431+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:52.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:52.431+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:52.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:52.432+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:53.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:53.079+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:53.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:53.079+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:53.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:53.079+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:53.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:53.080+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:53.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:52.992+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:53.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:52.992+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:53.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:52.992+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:53.410 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:52.993+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:53.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:53.455+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:53.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:53.455+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:53.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:53.455+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:53.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:53.456+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:54.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:54.122+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:54.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:54.122+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:54.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:54.122+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:54.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:54.122+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:54.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:53.998+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:54.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:53.998+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:54.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:53.998+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:54.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:53.999+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:54.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:54.462+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:54.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:54.462+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:54.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:54.462+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:54.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:54.463+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:55.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:55.021+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:55.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:55.021+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:55.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:55.021+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:55.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:55.022+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:55.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:55.120+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:55.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:55.120+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:55.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:55.120+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:55.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:55.121+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:55.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:55.428+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:55.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:55.428+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:55.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:55.428+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:55.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:55.429+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:56.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:56.024+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:56.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:56.024+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:56.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:56.024+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:56.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:56.025+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:56.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:56.169+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:56.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:56.169+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:56.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:56.169+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:56.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:56.170+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:56.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:56.449+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:56.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:56.449+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:56.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:56.449+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:56.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:56.450+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:57.383 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:31:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:31:57.023+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 30512 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:31:57.383 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:57.022+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:57.384 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:57.022+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:57.384 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:57.023+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:57.384 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:57.026+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:57.384 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:57.172+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:57.384 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:57.172+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:57.384 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:57.172+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:57.385 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:57.173+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:57.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:57.463+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:57.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:57.463+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:57.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:57.463+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:57.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:57.464+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:58.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:58.039+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:58.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:58.039+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:58.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:58.039+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:58.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:58.040+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:58.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:58.131+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:58.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:58.131+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:58.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:58.131+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:58.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:58.132+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:58.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:58.512+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:58.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:58.512+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:58.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:58.512+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:58.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:58.512+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:31:59.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:59.017+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:59.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:59.017+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:59.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:59.017+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:31:59.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:31:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:31:59.018+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:59.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:59.177+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:59.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:59.177+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:59.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:59.177+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:31:59.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:31:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:31:59.178+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:31:59.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:59.501+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:59.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:59.501+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:59.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:59.501+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:31:59.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:31:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:31:59.502+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:00.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:00.015+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:00.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:00.015+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:00.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:00.015+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:00.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:00.016+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:00.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:00.148+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:00.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:00.148+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:00.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:00.148+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:00.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:00.149+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:00.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:00.506+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:00.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:00.506+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:00.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:00.506+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:00.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:00.507+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:01.358 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:01.127+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:01.358 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:01.127+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:01.358 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:01.127+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:01.358 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:01.128+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:01.359 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:01.045+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:01.359 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:01.045+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:01.359 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:01.045+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:01.359 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:01.046+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:01.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:01.486+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:01.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:01.486+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:01.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:01.486+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:01.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:01.488+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:02.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:32:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:32:02.027+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 30593 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:32:02.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:02.053+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:02.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:02.053+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:02.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:02.053+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:02.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:02.054+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:02.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:02.115+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:02.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:02.115+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:02.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:02.115+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:02.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:02.116+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:02.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:02.504+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:02.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:02.504+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:02.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:02.504+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:02.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:02.505+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:03.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:03.158+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:03.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:03.158+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:03.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:03.158+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:03.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:03.158+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:03.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:03.013+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:03.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:03.013+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:03.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:03.013+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:03.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:03.013+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:03.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:03.492+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:03.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:03.492+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:03.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:03.492+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:03.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:03.492+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:04.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:04.053+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:04.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:04.053+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:04.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:04.053+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:04.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:04.054+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:04.160+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:04.160+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:04.160+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:04.161+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:04.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:04.480+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:04.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:04.480+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:04.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:04.480+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:04.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:04.480+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:05.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:05.009+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:05.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:05.009+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:05.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:05.009+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:05.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:05.011+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:05.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:05.201+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:05.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:05.201+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:05.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:05.201+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:05.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:05.201+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:05.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:05.502+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:05.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:05.502+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:05.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:05.502+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:05.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:05.503+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:06.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:05.989+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:06.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:05.989+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:06.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:05.989+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:06.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:05.990+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:06.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:06.194+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:06.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:06.194+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:06.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:06.194+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:06.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:06.196+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:06.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:06.460+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:06.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:06.460+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:06.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:06.460+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:06.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:06.461+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:07.241 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:32:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:32:07.029+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 30670 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:32:07.242 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:06.983+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:07.242 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:06.983+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:07.242 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:06.983+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:07.242 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:06.984+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:07.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:07.475+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:07.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:07.475+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:07.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:07.475+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:07.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:07.476+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:07.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:07.237+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:07.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:07.237+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:07.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:07.237+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:07.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:07.238+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:08.231 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:07.966+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:08.231 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:07.966+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:08.232 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:07.966+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:08.232 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:07.967+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:08.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:08.226+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:08.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:08.226+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:08.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:08.226+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:08.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:08.227+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:08.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:08.472+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:08.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:08.472+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:08.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:08.472+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:08.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:08.472+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:09.272 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:08.959+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:09.272 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:08.959+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:09.272 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:08.959+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:09.273 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:08.960+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:09.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:09.268+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:09.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:09.268+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:09.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:09.268+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:09.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:09.269+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:09.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:09.454+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:09.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:09.454+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:09.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:09.454+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:09.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:09.455+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:10.228 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:09.939+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:10.228 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:09.939+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:10.228 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:09.939+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:10.228 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:09.940+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:10.494 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:10.230+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:10.494 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:10.230+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:10.495 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:10.230+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:10.495 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:10.231+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:10.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:10.489+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:10.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:10.489+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:10.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:10.489+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:10.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:10.490+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:11.253 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:10.983+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:11.253 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:10.983+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:11.253 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:10.983+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:11.253 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:10.984+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:11.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:11.496+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:11.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:11.496+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:11.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:11.496+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:11.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:11.496+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:11.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:11.248+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:11.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:11.248+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:11.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:11.248+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:11.660 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:11.249+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:12.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:32:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:32:12.033+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 30748 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:32:12.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:12.255+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:12.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:12.255+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:12.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:12.255+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:12.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:12.256+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:12.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:12.034+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:12.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:12.034+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:12.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:12.034+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:12.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:12.035+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:12.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:12.487+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:12.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:12.487+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:12.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:12.487+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:12.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:12.488+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:13.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:13.020+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:13.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:13.020+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:13.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:13.020+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:13.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:13.021+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:13.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:13.258+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:13.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:13.258+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:13.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:13.258+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:13.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:13.258+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:13.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:13.475+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:13.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:13.475+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:13.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:13.475+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:13.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:13.476+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:14.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:14.038+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:14.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:14.038+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:14.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:14.038+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:14.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:14.039+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:14.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:14.234+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:14.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:14.235+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:14.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:14.235+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:14.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:14.235+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:14.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:14.472+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:14.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:14.472+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:14.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:14.472+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:14.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:14.473+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:15.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:15.040+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:15.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:15.040+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:15.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:15.040+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:15.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:15.041+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:15.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:15.238+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:15.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:15.238+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:15.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:15.238+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:15.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:15.239+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:15.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:15.505+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:15.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:15.505+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:15.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:15.505+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:15.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:15.505+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:16.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:16.078+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:16.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:16.078+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:16.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:16.078+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:16.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:16.079+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:16.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:16.226+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:16.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:16.226+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:16.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:16.226+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:16.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:16.227+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:16.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:16.515+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:16.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:16.515+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:16.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:16.515+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:16.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:16.517+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:17.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:32:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:32:17.037+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 30825 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:32:17.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:17.031+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:17.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:17.031+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:17.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:17.031+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:17.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:17.032+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:17.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:17.254+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:17.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:17.254+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:17.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:17.254+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:17.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:17.255+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:17.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:17.511+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:17.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:17.511+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:17.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:17.511+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:17.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:17.512+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:18.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:18.250+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:18.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:18.250+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:18.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:18.250+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:18.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:18.251+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:18.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:18.053+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:18.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:18.053+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:18.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:18.053+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:18.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:18.054+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:18.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:18.504+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:18.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:18.504+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:18.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:18.504+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:18.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:18.505+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:19.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:19.080+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:19.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:19.080+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:19.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:19.080+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:19.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:19.081+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:19.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:19.246+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:19.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:19.246+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:19.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:19.246+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:19.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:19.247+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:19.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:19.515+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:19.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:19.516+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:19.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:19.516+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:19.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:19.516+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:20.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:20.066+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:20.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:20.066+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:20.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:20.066+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:20.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:20.067+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:20.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:20.277+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:20.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:20.277+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:20.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:20.277+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:20.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:20.278+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:20.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:20.493+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:20.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:20.493+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:20.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:20.493+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:20.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:20.494+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:21.291 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:21.021+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:21.291 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:21.021+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:21.291 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:21.021+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:21.291 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:21.022+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:21.541 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:21.537+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:21.541 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:21.537+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:21.541 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:21.537+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:21.542 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:21.537+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:21.542 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:21.287+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:21.542 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:21.287+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:21.542 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:21.287+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:21.542 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:21.288+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:22.282 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:32:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:32:22.041+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 30903 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:32:22.283 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:21.993+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:22.283 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:21.993+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:22.283 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:21.993+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:22.283 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:21.994+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:22.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:22.278+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:22.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:22.278+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:22.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:22.278+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:22.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:22.279+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:22.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:22.515+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:22.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:22.515+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:22.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:22.515+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:22.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:22.515+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:23.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:22.987+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:23.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:22.987+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:23.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:22.987+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:23.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:22.988+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:23.498 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:23.243+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:23.498 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:23.243+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:23.498 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:23.243+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:23.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:23.244+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:23.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:23.504+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:23.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:23.504+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:23.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:23.504+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:23.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:23.505+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:24.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:23.991+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:24.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:23.991+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:24.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:23.991+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:24.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:23.991+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:24.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:24.214+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:24.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:24.214+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:24.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:24.214+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:24.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:24.215+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:24.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:24.490+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:24.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:24.490+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:24.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:24.490+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:24.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:24.491+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:25.202 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:24.944+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:25.202 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:24.944+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:25.202 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:24.944+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:25.203 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:24.944+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:25.515 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:25.198+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:25.515 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:25.198+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:25.515 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:25.198+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:25.515 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:25.199+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:25.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:25.511+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:25.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:25.511+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:25.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:25.511+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:25.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:25.512+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:26.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:26.190+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:26.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:26.190+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:26.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:26.190+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:26.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:26.190+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:26.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:25.989+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:26.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:25.989+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:26.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:25.989+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:26.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:25.990+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:26.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:26.529+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:26.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:26.529+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:26.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:26.529+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:26.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:26.529+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:27.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:32:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:32:27.044+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 30980 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:32:27.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:27.153+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:27.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:27.153+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:27.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:27.153+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:27.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:27.154+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:27.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:27.005+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:27.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:27.005+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:27.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:27.005+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:27.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:27.005+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:27.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:27.554+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:27.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:27.554+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:27.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:27.554+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:27.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:27.555+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:28.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:27.991+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:28.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:27.991+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:28.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:27.991+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:28.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:27.992+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:28.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:28.143+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:28.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:28.143+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:28.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:28.143+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:28.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:28.143+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:28.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:28.537+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:28.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:28.537+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:28.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:28.537+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:28.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:28.539+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:29.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:29.006+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:29.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:29.006+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:29.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:29.006+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:29.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:29.007+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:29.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:29.152+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:29.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:29.152+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:29.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:29.152+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:29.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:29.152+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:29.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:29.495+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:29.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:29.495+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:29.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:29.495+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:29.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:29.496+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:30.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:29.999+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:30.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:29.999+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:30.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:29.999+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:30.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:29.999+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:30.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:30.144+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:30.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:30.144+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:30.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:30.144+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:30.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:30.145+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:30.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:30.519+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:30.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:30.519+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:30.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:30.519+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:30.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:30.520+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:31.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:31.104+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:31.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:31.104+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:31.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:31.104+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:31.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:31.105+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:31.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:30.994+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:31.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:30.994+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:31.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:30.994+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:31.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:30.994+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:31.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:31.552+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:31.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:31.552+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:31.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:31.552+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:31.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:31.553+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:32.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:32:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:32:32.048+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 31061 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:32:32.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:32.026+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:32.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:32.026+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:32.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:32.026+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:32.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:32.026+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:32.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:32.089+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:32.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:32.089+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:32.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:32.089+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:32.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:32.090+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:32.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:32.538+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:32.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:32.538+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:32.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:32.538+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:32.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:32.539+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:33.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:33.033+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:33.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:33.033+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:33.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:33.033+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:33.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:33.034+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:33.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:33.065+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:33.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:33.065+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:33.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:33.065+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:33.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:33.065+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:33.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:33.498+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:33.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:33.498+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:33.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:33.498+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:33.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:33.499+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:34.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:34.091+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:34.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:34.091+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:34.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:34.091+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:34.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:34.092+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:34.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:34.058+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:34.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:34.058+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:34.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:34.058+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:34.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:34.059+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:34.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:34.465+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:34.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:34.465+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:34.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:34.465+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:34.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:34.466+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:35.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:35.041+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:35.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:35.041+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:35.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:35.041+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:35.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:35.042+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:35.076+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:35.077+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:35.077+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:35.077+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:35.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:35.419+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:35.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:35.419+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:35.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:35.419+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:35.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:35.420+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:36.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:36.072+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:36.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:36.072+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:36.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:36.072+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:36.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:36.073+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:36.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:36.092+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:36.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:36.092+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:36.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:36.092+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:36.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:36.093+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:36.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:36.445+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:36.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:36.445+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:36.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:36.445+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:36.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:36.446+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:37.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:32:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:32:37.052+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 31141 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:32:37.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:37.074+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:37.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:37.074+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:37.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:37.074+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:37.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:37.075+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:37.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:37.035+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:37.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:37.035+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:37.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:37.035+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:37.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:37.036+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:37.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:37.433+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:37.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:37.433+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:37.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:37.433+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:37.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:37.434+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:38.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:38.060+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:38.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:38.060+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:38.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:38.060+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:38.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:38.061+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:38.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:38.064+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:38.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:38.064+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:38.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:38.064+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:38.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:38.065+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:38.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:38.481+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:38.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:38.481+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:38.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:38.481+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:38.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:38.482+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:39.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:39.090+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:39.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:39.090+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:39.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:39.090+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:39.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:39.091+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:39.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:39.079+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:39.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:39.079+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:39.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:39.079+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:39.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:39.080+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:39.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:39.464+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:39.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:39.464+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:39.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:39.464+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:39.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:39.465+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:40.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:40.050+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:40.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:40.050+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:40.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:40.050+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:40.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:40.051+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:40.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:40.065+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:40.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:40.065+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:40.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:40.065+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:40.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:40.066+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:40.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:40.497+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:40.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:40.497+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:40.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:40.497+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:40.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:40.498+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:41.293 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:41.034+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:41.294 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:41.034+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:41.294 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:41.034+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:41.294 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:41.035+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:41.294 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:41.070+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:41.294 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:41.070+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:41.295 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:41.070+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:41.295 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:41.071+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:41.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:41.548+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:41.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:41.549+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:41.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:41.549+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:41.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:41.549+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:42.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:32:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:32:42.055+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 31225 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:32:42.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:41.995+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:42.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:41.995+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:42.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:41.995+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:42.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:41.996+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:42.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:42.023+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:42.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:42.023+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:42.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:42.023+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:42.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:42.023+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:42.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:42.515+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:42.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:42.515+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:42.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:42.515+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:42.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:42.516+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:43.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:43.002+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:43.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:43.002+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:43.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:43.002+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:43.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:43.004+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:43.061+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:43.061+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:43.061+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:43.062+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:43.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:43.505+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:43.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:43.505+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:43.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:43.505+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:43.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:43.505+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:44.366 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:44.048+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:44.366 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:44.048+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:44.366 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:44.048+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:44.366 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:44.049+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:44.366 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:44.100+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:44.367 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:44.100+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:44.367 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:44.100+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:44.367 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:44.101+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:44.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:44.519+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:44.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:44.519+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:44.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:44.519+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:44.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:44.520+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:45.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:45.085+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:45.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:45.085+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:45.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:45.085+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:45.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:45.086+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:45.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:45.095+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:45.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:45.095+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:45.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:45.095+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:45.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:45.095+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:45.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:45.470+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:45.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:45.470+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:45.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:45.470+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:45.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:45.471+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:46.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:46.137+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:46.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:46.137+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:46.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:46.137+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:46.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:46.138+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:46.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:46.128+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:46.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:46.128+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:46.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:46.128+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:46.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:46.129+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:46.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:46.487+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:46.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:46.487+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:46.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:46.487+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:46.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:46.488+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:47.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:32:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:32:47.059+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 31308 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:32:47.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:47.168+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:47.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:47.168+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:47.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:47.168+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:47.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:47.169+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:47.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:47.100+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:47.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:47.100+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:47.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:47.100+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:47.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:47.101+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:47.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:47.517+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:47.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:47.517+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:47.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:47.517+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:47.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:47.518+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:48.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:48.080+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:48.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:48.080+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:48.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:48.080+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:48.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:48.081+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:48.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:48.173+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:48.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:48.173+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:48.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:48.173+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:48.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:48.173+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:48.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:48.532+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:48.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:48.532+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:48.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:48.532+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:48.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:48.533+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:49.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:49.087+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:49.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:49.087+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:49.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:49.087+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:49.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:49.087+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:49.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:49.168+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:49.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:49.168+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:49.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:49.168+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:49.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:49.168+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:49.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:49.547+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:49.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:49.547+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:49.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:49.547+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:49.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:49.547+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:50.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:50.160+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:50.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:50.160+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:50.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:50.160+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:50.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:50.161+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:50.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:50.099+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:50.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:50.099+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:50.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:50.099+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:50.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:50.099+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:50.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:50.588+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:50.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:50.588+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:50.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:50.588+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:50.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:50.589+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:51.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:51.123+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:51.644 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:51.123+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:51.644 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:51.123+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:51.644 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:51.124+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:51.645 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:51.162+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:51.645 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:51.162+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:51.645 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:51.162+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:51.645 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:51.163+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:51.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:51.572+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:51.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:51.572+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:51.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:51.572+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:51.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:51.572+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:52.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:32:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:32:52.063+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 31392 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:32:52.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:52.086+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:52.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:52.086+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:52.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:52.086+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:52.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:52.087+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:52.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:52.141+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:52.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:52.141+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:52.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:52.141+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:52.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:52.142+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:52.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:52.612+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:52.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:52.612+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:52.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:52.612+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:52.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:52.612+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:53.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:53.190+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:53.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:53.190+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:53.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:53.190+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:53.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:53.191+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:53.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:53.072+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:53.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:53.072+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:53.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:53.072+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:53.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:53.073+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:53.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:53.569+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:53.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:53.569+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:53.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:53.569+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:53.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:53.570+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:54.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:54.214+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:54.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:54.214+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:54.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:54.214+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:54.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:54.215+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:54.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:54.076+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:54.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:54.076+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:54.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:54.076+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:54.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:54.077+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:54.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:54.577+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:54.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:54.577+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:54.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:54.577+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:54.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:54.578+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:55.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:55.206+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:55.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:55.206+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:55.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:55.206+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:55.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:55.207+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:55.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:55.037+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:55.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:55.037+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:55.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:55.037+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:55.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:55.038+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:55.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:55.587+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:55.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:55.587+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:55.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:55.587+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:55.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:55.588+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:56.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:56.196+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:56.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:56.196+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:56.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:56.196+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:56.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:56.197+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:56.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:56.021+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:56.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:56.021+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:56.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:56.021+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:56.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:56.022+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:56.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:56.587+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:56.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:56.587+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:56.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:56.587+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:56.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:56.588+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:57.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:32:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:32:57.066+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 31475 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:32:57.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:57.031+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:57.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:57.031+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:57.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:57.031+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:57.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:57.033+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:57.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:57.220+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:57.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:57.220+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:57.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:57.220+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:57.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:57.220+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:57.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:57.607+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:57.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:57.607+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:57.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:57.607+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:57.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:57.608+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:58.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:58.020+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:58.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:58.020+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:58.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:58.020+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:58.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:58.021+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:58.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:58.238+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:58.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:58.238+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:58.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:58.238+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:58.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:58.239+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:58.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:58.609+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:58.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:58.609+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:58.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:58.609+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:58.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:58.610+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:32:59.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:59.256+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:59.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:59.256+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:59.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:59.256+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:32:59.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:32:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:32:59.257+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:59.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:59.070+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:59.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:59.070+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:59.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:59.070+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:32:59.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:32:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:32:59.071+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:32:59.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:59.582+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:59.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:59.582+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:59.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:59.582+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:32:59.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:32:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:32:59.583+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:00.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:00.284+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:00.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:00.284+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:00.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:00.284+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:00.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:00.284+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:00.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:00.119+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:00.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:00.119+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:00.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:00.119+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:00.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:00.120+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:00.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:00.630+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:00.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:00.630+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:00.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:00.630+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:00.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:00.631+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:01.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:01.148+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:01.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:01.148+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:01.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:01.148+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:01.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:01.149+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:01.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:01.264+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:01.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:01.264+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:01.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:01.264+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:01.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:01.265+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:02.077 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:01.660+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:02.077 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:01.660+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:02.078 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:01.660+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:02.078 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:01.661+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:02.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:33:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:33:02.070+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 31557 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:33:02.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:02.134+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:02.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:02.134+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:02.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:02.134+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:02.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:02.135+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:02.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:02.303+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:02.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:02.303+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:02.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:02.303+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:02.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:02.304+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:03.110 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:02.694+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:03.111 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:02.694+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:03.111 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:02.694+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:03.111 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:02.695+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:03.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:03.107+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:03.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:03.107+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:03.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:03.107+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:03.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:03.108+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:03.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:03.264+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:03.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:03.264+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:03.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:03.264+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:03.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:03.265+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:03.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:03.650+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:03.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:03.650+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:03.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:03.650+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:03.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:03.651+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:04.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:04.117+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:04.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:04.117+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:04.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:04.117+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:04.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:04.117+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:04.226+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:04.226+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:04.226+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:04.227+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:04.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:04.637+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:04.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:04.637+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:04.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:04.637+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:04.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:04.638+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:05.647 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:05.246+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:05.648 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:05.246+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:05.648 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:05.246+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:05.648 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:05.247+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:05.648 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:05.164+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:05.648 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:05.164+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:05.649 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:05.164+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:05.649 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:05.165+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:05.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:05.644+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:05.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:05.644+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:05.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:05.644+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:05.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:05.645+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:06.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:06.286+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:06.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:06.286+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:06.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:06.286+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:06.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:06.287+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:06.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:06.147+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:06.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:06.147+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:06.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:06.147+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:06.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:06.148+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:07.078 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:06.663+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:07.078 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:06.663+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:07.078 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:06.663+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:07.078 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:06.664+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:07.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:33:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:33:07.074+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 31640 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:33:07.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:07.246+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:07.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:07.246+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:07.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:07.246+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:07.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:07.247+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:07.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:07.183+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:07.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:07.183+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:07.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:07.183+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:07.409 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:07.183+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:08.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:07.679+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:08.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:07.679+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:08.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:07.679+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:08.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:07.679+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:08.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:08.295+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:08.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:08.295+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:08.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:08.295+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:08.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:08.295+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:08.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:08.212+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:08.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:08.212+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:08.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:08.212+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:08.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:08.213+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:09.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:08.728+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:09.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:08.728+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:09.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:08.728+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:09.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:08.729+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:09.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:09.172+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:09.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:09.172+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:09.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:09.172+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:09.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:09.173+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:09.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:09.292+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:09.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:09.292+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:09.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:09.292+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:09.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:09.293+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:10.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:09.720+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:10.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:09.720+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:10.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:09.720+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:10.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:09.721+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:10.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:10.180+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:10.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:10.180+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:10.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:10.180+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:10.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:10.180+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:10.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:10.340+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:10.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:10.340+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:10.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:10.340+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:10.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:10.341+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:11.100 ERROR:paramiko.transport:Socket exception: No route to host (113) 2024-02-09T22:33:11.106 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T22:33:11.106 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-02-09T22:33:11.107 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-02-09T22:33:11.124 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-09T22:33:11.136 INFO:tasks.cephadm:Teardown begin 2024-02-09T22:33:11.136 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 1845, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-02-09T22:33:11.141 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-09T22:33:11.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:10.709+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:11.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:10.709+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:11.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:10.709+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:11.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:10.709+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:11.178 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:33:11.178 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:33:11.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:11.214+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:11.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:11.214+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:11.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:11.214+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:11.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:11.214+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:11.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:11.334+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:11.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:11.334+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:11.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:11.334+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:11.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:11.335+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:12.013 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:11.687+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:12.014 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:11.687+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:12.014 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:11.687+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:12.014 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:11.687+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:12.304 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:33:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:33:12.078+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 31723 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:33:12.304 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:12.191+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:12.304 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:12.191+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:12.304 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:12.191+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:12.305 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:12.192+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:12.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:12.300+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:12.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:12.300+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:12.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:12.300+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:12.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:12.301+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:13.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:12.705+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:13.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:12.705+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:13.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:12.705+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:13.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:12.706+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:13.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:13.195+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:13.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:13.195+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:13.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:13.195+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:13.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:13.196+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:13.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:13.282+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:13.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:13.282+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:13.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:13.282+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:13.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:13.283+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:13.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:13.739+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:13.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:13.739+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:13.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:13.739+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:13.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:13.740+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:14.236 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:33:14.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:14.165+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:14.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:14.165+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:14.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:14.165+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:14.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:14.165+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:14.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:14.298+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:14.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:14.298+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:14.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:14.298+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:14.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:14.299+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:15.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:14.725+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:15.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:14.725+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:15.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:14.725+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:15.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:14.726+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:15.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:15.154+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:15.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:15.154+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:15.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:15.154+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:15.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:15.154+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:15.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:15.331+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:15.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:15.331+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:15.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:15.331+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:15.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:15.332+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:16.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:15.695+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:16.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:15.695+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:16.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:15.695+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:16.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:15.696+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:16.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:16.184+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:16.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:16.184+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:16.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:16.184+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:16.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:16.185+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:16.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:16.340+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:16.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:16.340+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:16.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:16.340+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:16.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:16.341+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:17.085 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:16.683+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:17.086 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:16.683+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:17.086 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:16.683+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:17.086 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:16.683+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:17.348 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:33:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:33:17.081+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 31805 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:33:17.349 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:17.218+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:17.349 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:17.218+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:17.349 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:17.218+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:17.349 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:17.218+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:17.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:17.345+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:17.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:17.345+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:17.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:17.345+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:17.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:17.346+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:18.096 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:17.715+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:18.097 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:17.715+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:18.097 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:17.715+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:18.097 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:17.716+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:18.382 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:18.253+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:18.383 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:18.253+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:18.383 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:18.253+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:18.383 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:18.254+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:18.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:18.385+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:18.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:18.385+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:18.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:18.385+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:18.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:18.386+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:19.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:18.703+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:19.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:18.703+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:19.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:18.703+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:19.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:18.704+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:19.510 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:19.255+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:19.510 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:19.255+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:19.511 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:19.255+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:19.511 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:19.256+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:19.511 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:19.369+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:19.511 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:19.369+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:19.511 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:19.369+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:19.512 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:19.370+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:19.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:19.701+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:19.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:19.701+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:19.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:19.701+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:19.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:19.702+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:20.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:20.219+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:20.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:20.219+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:20.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:20.219+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:20.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:20.220+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:20.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:20.409+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:20.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:20.409+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:20.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:20.409+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:20.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:20.410+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:21.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:20.679+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:21.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:20.679+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:21.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:20.679+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:21.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:20.680+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:21.649 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:21.173+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:21.649 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:21.173+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:21.649 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:21.173+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:21.649 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:21.174+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:21.649 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:21.360+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:21.650 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:21.360+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:21.650 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:21.360+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:21.650 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:21.361+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:21.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:21.645+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:21.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:21.645+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:21.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:21.645+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:21.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:21.646+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:22.379 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:33:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:33:22.085+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 31889 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:33:22.379 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:22.166+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:22.379 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:22.166+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:22.380 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:22.166+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:22.380 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:22.166+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:22.638 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:22.375+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:22.639 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:22.375+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:22.639 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:22.375+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:22.639 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:22.376+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:22.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:22.635+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:22.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:22.635+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:22.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:22.635+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:22.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:22.636+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:23.239 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:33:23.240 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:33:23.398 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:23.117+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:23.399 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:23.117+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:23.399 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:23.117+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:23.399 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:23.118+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:23.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:23.623+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:23.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:23.623+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:23.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:23.623+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:23.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:23.624+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:23.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:23.394+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:23.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:23.394+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:23.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:23.394+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:23.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:23.396+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:24.371 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:24.115+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:24.371 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:24.115+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:24.372 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:24.115+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:24.372 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:24.116+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:24.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:24.595+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:24.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:24.595+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:24.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:24.595+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:24.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:24.596+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:24.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:24.367+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:24.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:24.367+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:24.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:24.367+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:24.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:24.368+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:25.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:25.151+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:25.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:25.151+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:25.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:25.151+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:25.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:25.152+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:25.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:25.360+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:25.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:25.360+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:25.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:25.360+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:25.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:25.361+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:25.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:25.551+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:25.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:25.551+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:25.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:25.551+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:25.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:25.552+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:26.300 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:33:26.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:26.131+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:26.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:26.131+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:26.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:26.131+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:26.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:26.133+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:26.355+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:26.355+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:26.355+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:26.356+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:26.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:26.533+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:26.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:26.533+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:26.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:26.533+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:26.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:26.534+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:27.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:33:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:33:27.089+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 31971 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:33:27.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:27.162+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:27.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:27.162+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:27.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:27.162+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:27.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:27.163+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:27.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:27.336+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:27.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:27.336+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:27.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:27.336+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:27.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:27.337+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:27.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:27.509+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:27.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:27.509+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:27.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:27.509+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:27.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:27.509+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:28.483 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:28.172+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:28.484 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:28.172+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:28.484 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:28.172+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:28.484 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:28.173+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:28.484 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:28.348+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:28.484 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:28.348+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:28.485 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:28.348+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:28.485 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:28.349+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:28.886 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:28.479+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:28.886 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:28.479+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:28.886 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:28.479+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:28.886 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:28.480+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:29.517 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:29.397+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:29.517 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:29.397+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:29.517 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:29.397+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:29.518 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:29.398+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:29.518 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:29.169+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:29.518 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:29.169+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:29.518 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:29.169+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:29.518 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:29.169+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:29.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:29.513+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:29.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:29.513+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:29.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:29.513+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:29.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:29.514+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:30.445 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:30.177+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:30.446 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:30.177+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:30.446 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:30.177+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:30.446 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:30.178+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:30.446 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:30.392+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:30.446 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:30.392+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:30.447 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:30.392+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:30.447 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:30.393+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:30.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:30.552+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:30.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:30.552+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:30.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:30.552+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:30.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:30.553+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:31.546 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:31.180+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:31.547 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:31.180+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:31.547 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:31.180+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:31.547 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:31.181+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:31.547 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:31.368+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:31.547 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:31.368+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:31.548 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:31.368+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:31.548 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:31.368+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:31.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:31.543+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:31.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:31.543+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:31.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:31.543+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:31.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:31.544+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:32.374 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:33:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:33:32.093+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 32045 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:33:32.375 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:32.209+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:32.375 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:32.209+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:32.375 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:32.209+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:32.376 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:32.210+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:32.376 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:32.371+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:32.376 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:32.371+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:32.376 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:32.371+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:32.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:32.372+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:32.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:32.586+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:32.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:32.586+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:32.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:32.586+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:32.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:32.587+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:33.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:33.196+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:33.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:33.196+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:33.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:33.196+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:33.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:33.197+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:33.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:33.355+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:33.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:33.355+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:33.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:33.355+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:33.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:33.355+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:33.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:33.545+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:33.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:33.545+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:33.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:33.545+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:33.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:33.546+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:34.567 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:34.195+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:34.568 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:34.195+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:34.568 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:34.195+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:34.568 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:34.196+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:34.568 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:34.368+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:34.569 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:34.368+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:34.569 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:34.368+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:34.569 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:34.369+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:34.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:34.564+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:34.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:34.564+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:34.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:34.564+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:34.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:34.565+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:35.553 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:35.201+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:35.554 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:35.201+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:35.554 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:35.201+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:35.554 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:35.202+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:35.554 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:35.349+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:35.554 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:35.349+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:35.554 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:35.349+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:35.555 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:35.350+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:35.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:35.549+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:35.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:35.549+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:35.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:35.549+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:35.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:35.550+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:36.586 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:36.185+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:36.586 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:36.185+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:36.587 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:36.185+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:36.587 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:36.186+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:36.587 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:36.311+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:36.587 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:36.311+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:36.587 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:36.311+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:36.587 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:36.312+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:36.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:36.582+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:36.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:36.582+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:36.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:36.582+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:36.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:36.583+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:37.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:33:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:33:37.097+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 32127 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:33:37.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:37.163+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:37.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:37.163+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:37.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:37.163+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:37.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:37.163+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:37.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:37.315+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:37.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:37.315+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:37.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:37.315+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:37.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:37.315+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:37.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:37.549+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:37.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:37.549+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:37.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:37.549+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:37.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:37.550+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:38.301 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:33:38.302 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:33:38.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:38.119+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:38.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:38.119+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:38.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:38.119+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:38.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:38.120+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:38.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:38.364+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:38.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:38.364+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:38.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:38.364+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:38.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:38.364+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:38.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:38.581+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:38.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:38.581+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:38.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:38.581+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:38.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:38.582+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:39.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:39.117+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:39.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:39.117+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:39.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:39.117+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:39.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:39.118+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:39.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:39.345+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:39.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:39.345+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:39.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:39.345+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:39.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:39.346+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:39.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:39.556+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:39.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:39.556+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:39.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:39.556+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:39.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:39.557+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:40.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:40.126+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:40.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:40.126+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:40.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:40.126+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:40.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:40.127+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:40.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:40.315+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:40.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:40.315+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:40.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:40.315+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:40.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:40.316+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:40.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:40.603+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:40.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:40.603+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:40.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:40.603+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:40.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:40.604+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:41.372 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:33:41.652 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:41.163+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:41.652 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:41.163+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:41.653 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:41.163+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:41.653 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:41.164+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:41.653 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:41.311+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:41.653 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:41.311+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:41.653 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:41.311+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:41.653 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:41.312+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:41.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:41.648+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:41.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:41.648+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:41.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:41.648+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:41.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:41.649+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:42.356 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:33:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:33:42.101+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 32211 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:33:42.356 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:42.351+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:42.357 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:42.351+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:42.357 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:42.351+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:42.357 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:42.352+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:42.357 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:42.124+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:42.357 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:42.124+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:42.357 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:42.125+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:42.358 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:42.125+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:43.147 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:42.664+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:43.147 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:42.664+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:43.147 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:42.664+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:43.147 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:42.665+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:43.401 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:43.139+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:43.401 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:43.139+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:43.401 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:43.139+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:43.401 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:43.140+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:43.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:43.396+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:43.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:43.396+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:43.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:43.396+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:43.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:43.396+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:44.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:43.657+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:44.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:43.657+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:44.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:43.657+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:44.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:43.658+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:44.372 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:33:44.373 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:33:44.444 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:33:44.444 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-09T22:33:44.445 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-09T22:33:44.445 DEBUG:teuthology.orchestra.run.smithi006:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-09T22:33:44.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:33:44.464 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:33:44.622 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:44.180+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:44.622 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:44.180+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:44.622 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:44.180+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:44.622 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:44.181+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:44.622 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:44.348+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:44.623 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:44.348+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:44.623 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:44.348+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:44.623 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:44.349+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:44.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:44.617+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:44.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:44.617+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:44.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:44.617+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:44.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:44.618+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:45.591 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:45.210+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:45.591 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:45.210+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:45.591 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:45.210+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:45.591 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:45.211+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:45.592 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:45.342+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:45.592 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:45.342+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:45.592 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:45.342+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:45.592 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:45.343+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:45.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:45.586+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:45.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:45.586+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:45.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:45.586+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:45.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:45.587+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:46.592 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:46.225+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:46.593 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:46.225+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:46.593 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:46.225+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:46.593 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:46.226+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:46.593 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:46.359+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:46.593 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:46.359+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:46.594 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:46.359+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:46.594 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:46.360+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:46.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:46.588+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:46.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:46.588+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:46.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:46.588+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:46.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:46.589+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:47.367 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:33:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:33:47.104+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 32294 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:33:47.368 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:47.206+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:47.368 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:47.206+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:47.368 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:47.206+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:47.368 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:47.207+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:47.516 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:33:47.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:47.583+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:47.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:47.583+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:47.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:47.583+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:47.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:47.583+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:47.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:47.360+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:47.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:47.360+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:47.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:47.360+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:47.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:47.361+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:48.627 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:48.238+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:48.627 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:48.238+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:48.628 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:48.238+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:48.628 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:48.239+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:48.628 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:48.319+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:48.628 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:48.319+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:48.628 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:48.319+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:48.628 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:48.319+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:48.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:48.622+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:48.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:48.622+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:48.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:48.622+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:48.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:48.623+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:49.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:49.195+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:49.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:49.196+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:49.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:49.196+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:49.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:49.197+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:49.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:49.338+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:49.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:49.338+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:49.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:49.338+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:49.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:49.339+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:50.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:49.672+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:50.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:49.672+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:50.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:49.672+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:50.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:49.673+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:50.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:50.158+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:50.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:50.158+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:50.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:50.158+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:50.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:50.159+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:50.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:50.389+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:50.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:50.389+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:50.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:50.389+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:50.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:50.390+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:51.154 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:50.669+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:51.155 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:50.669+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:51.155 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:50.669+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:51.155 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:50.670+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:51.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:51.150+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:51.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:51.150+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:51.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:51.150+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:51.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:51.150+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:51.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:51.391+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:51.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:51.391+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:51.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:51.391+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:51.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:51.391+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:52.113 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:51.701+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:52.114 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:51.701+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:52.114 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:51.701+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:52.114 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:51.701+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:52.403 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:52.116+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:52.403 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:52.116+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:52.403 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:52.116+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:52.403 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:52.117+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:52.404 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:33:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:33:52.109+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 32378 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:33:52.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:52.398+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:52.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:52.398+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:52.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:52.398+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:52.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:52.399+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:53.120 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:52.666+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:53.120 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:52.666+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:53.121 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:52.666+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:53.121 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:52.667+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:53.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:53.116+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:53.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:53.116+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:53.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:53.116+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:53.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:53.117+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:53.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:53.441+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:53.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:53.441+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:53.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:53.441+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:53.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:53.442+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:53.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:53.682+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:53.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:53.682+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:53.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:53.682+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:53.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:53.683+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:54.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:54.139+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:54.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:54.139+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:54.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:54.139+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:54.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:54.140+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:54.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:54.451+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:54.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:54.451+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:54.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:54.451+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:54.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:54.452+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:54.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:54.637+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:54.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:54.637+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:54.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:54.637+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:54.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:54.638+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:55.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:55.131+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:55.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:55.131+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:55.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:55.131+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:55.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:55.132+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:55.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:55.490+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:55.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:55.490+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:55.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:55.490+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:55.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:55.491+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:55.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:55.635+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:55.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:55.635+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:55.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:55.635+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:55.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:55.637+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:56.462 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:56.172+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:56.462 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:56.172+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:56.462 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:56.172+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:56.463 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:56.173+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:56.517 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:33:56.518 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:33:56.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:56.458+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:56.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:56.458+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:56.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:56.458+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:56.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:56.459+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:56.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:56.609+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:56.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:56.609+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:56.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:56.609+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:56.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:56.609+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:57.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:33:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:33:57.113+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 32462 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:33:57.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:57.131+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:57.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:57.131+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:57.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:57.131+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:57.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:57.132+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:57.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:57.647+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:57.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:57.647+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:57.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:57.647+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:57.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:57.648+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:57.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:57.411+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:57.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:57.411+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:57.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:57.411+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:57.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:57.412+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:58.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:58.153+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:58.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:58.153+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:58.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:58.153+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:58.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:58.154+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:58.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:58.413+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:58.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:58.413+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:58.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:58.413+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:58.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:58.414+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:58.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:58.645+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:58.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:58.645+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:58.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:58.645+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:58.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:58.647+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:33:59.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:59.114+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:59.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:59.114+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:59.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:59.114+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:33:59.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:33:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:33:59.115+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:59.584 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:33:59.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:59.457+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:59.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:59.457+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:59.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:59.457+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:33:59.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:33:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:33:59.458+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:33:59.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:59.598+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:59.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:59.598+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:59.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:59.598+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:33:59.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:33:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:33:59.599+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:00.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:00.066+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:00.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:00.066+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:00.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:00.066+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:00.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:00.066+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:00.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:00.419+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:00.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:00.419+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:00.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:00.419+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:00.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:00.420+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:00.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:00.587+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:00.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:00.587+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:00.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:00.587+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:00.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:00.588+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:01.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:01.063+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:01.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:01.063+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:01.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:01.063+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:01.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:01.064+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:01.766 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:01.578+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:01.767 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:01.578+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:01.767 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:01.578+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:01.767 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:01.579+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:01.767 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:01.410+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:01.768 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:01.410+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:01.768 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:01.410+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:01.768 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:01.411+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:02.372 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:34:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:34:02.117+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 32546 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:34:02.372 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:02.081+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:02.373 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:02.081+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:02.373 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:02.081+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:02.373 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:02.082+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:02.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:02.368+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:02.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:02.368+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:02.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:02.368+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:02.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:02.369+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:02.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:02.607+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:02.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:02.607+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:02.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:02.607+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:02.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:02.608+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:03.288 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:03.050+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:03.288 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:03.050+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:03.288 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:03.050+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:03.288 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:03.051+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:03.640 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:03.329+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:03.640 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:03.329+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:03.640 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:03.329+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:03.640 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:03.330+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:03.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:03.636+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:03.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:03.636+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:03.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:03.636+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:03.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:03.637+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:04.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:04.091+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:04.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:04.091+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:04.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:04.091+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:04.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:04.092+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:04.290+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:04.290+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:04.290+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:04.291+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:04.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:04.593+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:04.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:04.593+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:04.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:04.593+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:04.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:04.594+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:05.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:05.085+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:05.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:05.085+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:05.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:05.085+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:05.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:05.087+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:05.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:05.311+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:05.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:05.311+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:05.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:05.311+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:05.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:05.312+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:05.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:05.567+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:05.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:05.567+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:05.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:05.567+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:05.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:05.568+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:06.353 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:06.054+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:06.353 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:06.054+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:06.354 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:06.055+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:06.354 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:06.056+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:06.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:06.529+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:06.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:06.529+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:06.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:06.529+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:06.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:06.530+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:06.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:06.350+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:06.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:06.350+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:06.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:06.350+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:06.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:06.350+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:07.337 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:34:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:34:07.121+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 32629 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:34:07.338 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:07.069+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:07.338 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:07.069+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:07.338 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:07.069+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:07.338 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:07.070+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:07.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:07.334+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:07.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:07.334+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:07.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:07.334+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:07.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:07.334+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:07.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:07.572+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:07.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:07.572+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:07.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:07.572+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:07.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:07.573+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:08.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:08.337+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:08.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:08.337+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:08.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:08.337+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:08.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:08.338+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:08.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:08.095+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:08.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:08.095+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:08.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:08.095+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:08.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:08.095+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:08.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:08.588+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:08.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:08.588+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:08.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:08.588+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:08.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:08.589+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:09.340 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:09.072+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:09.341 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:09.072+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:09.341 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:09.072+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:09.341 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:09.073+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:09.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:09.545+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:09.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:09.545+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:09.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:09.545+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:09.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:09.546+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:09.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:09.337+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:09.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:09.337+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:09.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:09.337+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:09.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:09.338+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:10.381 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:10.078+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:10.382 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:10.078+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:10.382 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:10.078+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:10.382 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:10.079+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:10.382 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:10.301+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:10.382 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:10.301+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:10.382 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:10.301+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:10.383 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:10.302+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:10.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:10.502+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:10.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:10.502+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:10.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:10.502+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:10.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:10.503+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:11.293 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:11.076+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:11.293 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:11.076+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:11.293 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:11.076+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:11.293 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:11.077+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:11.585 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:34:11.586 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:34:11.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:11.289+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:11.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:11.289+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:11.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:11.289+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:11.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:11.290+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:11.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:11.530+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:11.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:11.530+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:11.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:11.530+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:11.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:11.532+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:12.253 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:34:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:34:12.125+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 32714 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:34:12.253 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:12.073+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:12.254 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:12.073+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:12.254 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:12.073+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:12.254 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:12.074+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:12.551 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:12.255+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:12.552 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:12.255+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:12.552 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:12.255+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:12.552 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:12.256+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:12.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:12.548+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:12.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:12.548+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:12.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:12.548+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:12.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:12.549+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:13.287 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:13.025+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:13.288 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:13.025+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:13.288 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:13.025+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:13.288 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:13.026+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:13.584 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:13.284+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:13.585 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:13.284+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:13.585 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:13.284+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:13.585 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:13.285+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:13.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:13.581+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:13.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:13.581+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:13.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:13.581+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:13.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:13.582+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:14.392 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:13.999+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:14.393 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:13.999+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:14.393 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:13.999+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:14.393 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:13.999+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:14.393 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:14.243+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:14.394 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:14.243+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:14.394 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:14.243+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:14.394 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:14.244+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:14.656 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:34:14.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:14.569+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:14.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:14.569+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:14.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:14.569+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:14.659 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:14.574+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:15.256 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:14.979+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:15.256 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:14.979+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:15.256 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:14.979+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:15.256 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:14.980+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:15.598 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:15.252+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:15.599 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:15.252+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:15.599 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:15.252+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:15.599 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:15.253+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:15.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:15.595+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:15.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:15.595+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:15.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:15.595+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:15.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:15.595+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:16.244 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:15.957+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:16.244 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:15.957+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:16.244 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:15.957+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:16.244 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:15.958+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:16.552 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:16.240+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:16.552 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:16.240+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:16.552 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:16.240+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:16.552 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:16.241+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:16.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:16.548+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:16.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:16.548+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:16.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:16.548+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:16.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:16.549+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:17.219 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:34:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:34:17.129+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 32797 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:34:17.219 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:16.945+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:17.219 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:16.945+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:17.220 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:16.945+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:17.220 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:16.946+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:17.557 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:17.215+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:17.558 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:17.215+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:17.558 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:17.215+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:17.558 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:17.216+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:17.657 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:34:17.658 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:34:17.724 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:34:17.724 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-09T22:34:17.724 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-09T22:34:17.727 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi/7554325/remote/smithi006/crash 2024-02-09T22:34:17.728 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/crash -- . 2024-02-09T22:34:17.794 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/lib/ceph/ee26d820-c794-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi/7554325/remote/smithi090/crash 2024-02-09T22:34:17.795 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:34:17.796 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:34:17.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:17.554+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:17.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:17.554+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:17.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:17.554+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:17.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:17.555+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:18.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:17.992+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:18.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:17.992+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:18.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:17.992+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:18.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:17.993+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:18.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:18.221+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:18.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:18.221+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:18.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:18.221+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:18.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:18.222+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:18.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:18.568+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:18.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:18.568+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:18.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:18.568+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:18.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:18.569+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:19.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:19.025+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:19.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:19.025+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:19.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:19.025+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:19.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:19.026+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:19.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:19.213+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:19.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:19.213+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:19.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:19.213+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:19.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:19.214+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:19.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:19.590+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:19.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:19.590+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:19.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:19.590+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:19.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:19.591+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:20.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:20.011+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:20.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:20.011+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:20.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:20.011+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:20.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:20.012+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:20.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:20.237+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:20.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:20.237+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:20.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:20.237+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:20.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:20.238+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:20.860 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:34:20.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:20.598+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:20.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:20.598+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:20.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:20.598+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:20.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:20.598+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:21.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:21.005+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:21.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:21.005+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:21.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:21.005+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:21.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:21.006+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:21.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:21.232+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:21.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:21.232+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:21.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:21.232+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:21.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:21.233+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:21.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:21.576+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:21.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:21.576+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:21.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:21.576+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:21.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:21.577+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:22.391 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:34:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:34:22.133+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 32882 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:34:22.391 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:22.043+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:22.391 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:22.043+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:22.391 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:22.043+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:22.392 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:22.044+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:22.392 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:22.196+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:22.392 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:22.196+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:22.392 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:22.196+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:22.392 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:22.197+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:22.642 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:22.616+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:22.643 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:22.616+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:22.643 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:22.616+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:22.643 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:22.616+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:23.244 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:23.061+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:23.244 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:23.061+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:23.244 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:23.061+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:23.245 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:23.061+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:23.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:23.240+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:23.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:23.240+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:23.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:23.240+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:23.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:23.242+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:23.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:23.616+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:23.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:23.616+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:23.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:23.616+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:23.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:23.617+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:24.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:24.077+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:24.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:24.077+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:24.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:24.077+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:24.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:24.078+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:24.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:24.278+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:24.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:24.278+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:24.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:24.278+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:24.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:24.279+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:24.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:24.568+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:24.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:24.568+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:24.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:24.568+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:24.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:24.569+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:25.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:25.103+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:25.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:25.103+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:25.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:25.103+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:25.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:25.104+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:25.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:25.313+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:25.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:25.313+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:25.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:25.313+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:25.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:25.313+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:25.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:25.547+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:25.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:25.547+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:25.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:25.547+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:25.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:25.548+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:26.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:26.088+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:26.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:26.088+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:26.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:26.088+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:26.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:26.089+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:26.329+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:26.329+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:26.329+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:26.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:26.330+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:26.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:26.582+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:26.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:26.582+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:26.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:26.582+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:26.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:26.583+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:27.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:34:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:34:27.137+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 32965 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:34:27.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:27.114+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:27.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:27.114+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:27.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:27.114+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:27.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:27.115+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:27.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:27.307+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:27.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:27.307+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:27.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:27.307+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:27.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:27.308+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:27.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:27.628+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:27.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:27.628+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:27.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:27.628+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:27.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:27.629+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:28.636 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:28.164+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:28.637 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:28.164+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:28.637 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:28.164+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:28.637 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:28.164+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:28.637 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:28.272+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:28.637 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:28.272+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:28.638 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:28.272+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:28.638 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:28.273+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:28.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:28.633+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:28.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:28.633+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:28.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:28.633+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:28.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:28.634+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:29.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:29.178+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:29.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:29.178+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:29.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:29.178+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:29.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:29.178+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:29.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:29.233+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:29.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:29.233+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:29.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:29.233+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:29.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:29.234+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:29.861 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:34:29.862 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:34:30.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:29.670+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:30.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:29.670+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:30.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:29.670+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:30.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:29.671+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:30.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:30.282+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:30.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:30.282+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:30.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:30.282+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:30.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:30.283+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:30.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:30.166+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:30.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:30.166+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:30.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:30.166+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:30.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:30.167+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:31.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:30.661+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:31.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:30.661+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:31.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:30.661+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:31.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:30.661+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:31.638 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:31.176+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:31.639 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:31.176+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:31.639 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:31.176+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:31.639 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:31.177+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:31.639 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:31.286+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:31.639 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:31.286+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:31.640 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:31.286+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:31.640 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:31.287+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:31.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:31.635+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:31.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:31.635+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:31.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:31.635+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:31.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:31.636+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:32.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:34:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:34:32.141+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 33050 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:34:32.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:32.142+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:32.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:32.142+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:32.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:32.142+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:32.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:32.143+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:32.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:32.276+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:32.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:32.276+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:32.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:32.276+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:32.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:32.277+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:32.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:32.612+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:32.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:32.612+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:32.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:32.612+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:32.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:32.613+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:32.924 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:34:33.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:33.145+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:33.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:33.145+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:33.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:33.145+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:33.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:33.146+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:33.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:33.293+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:33.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:33.293+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:33.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:33.293+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:33.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:33.294+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:33.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:33.650+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:33.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:33.650+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:33.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:33.650+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:33.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:33.651+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:34.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:34.142+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:34.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:34.142+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:34.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:34.142+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:34.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:34.143+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:34.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:34.342+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:34.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:34.342+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:34.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:34.342+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:34.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:34.343+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:35.113 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:34.682+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:35.113 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:34.682+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:35.113 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:34.682+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:35.113 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:34.682+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:35.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:35.109+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:35.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:35.109+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:35.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:35.109+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:35.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:35.110+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:35.332+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:35.332+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:35.332+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:35.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:35.333+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:36.116 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:35.693+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:36.116 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:35.693+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:36.116 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:35.693+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:36.116 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:35.694+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:36.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:36.298+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:36.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:36.298+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:36.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:36.298+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:36.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:36.299+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:36.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:36.112+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:36.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:36.112+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:36.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:36.112+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:36.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:36.113+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:37.081 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:36.678+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:37.082 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:36.678+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:37.082 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:36.678+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:37.082 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:36.679+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:37.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:34:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:34:37.145+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 33133 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:34:37.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:37.078+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:37.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:37.078+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:37.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:37.078+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:37.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:37.079+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:37.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:37.304+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:37.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:37.304+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:37.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:37.304+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:37.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:37.305+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:38.075 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:37.689+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:38.075 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:37.689+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:38.075 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:37.689+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:38.075 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:37.690+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:38.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:38.296+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:38.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:38.296+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:38.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:38.296+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:38.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:38.297+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:38.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:38.071+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:38.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:38.071+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:38.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:38.071+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:38.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:38.072+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:39.065 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:38.725+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:39.066 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:38.725+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:39.066 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:38.725+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:39.066 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:38.726+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:39.333 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:39.062+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:39.333 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:39.062+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:39.333 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:39.062+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:39.333 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:39.063+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:39.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:39.329+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:39.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:39.329+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:39.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:39.329+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:39.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:39.330+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:40.044 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:39.711+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:40.044 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:39.711+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:40.045 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:39.711+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:40.045 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:39.712+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:40.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:40.040+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:40.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:40.040+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:40.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:40.040+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:40.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:40.041+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:40.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:40.286+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:40.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:40.286+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:40.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:40.286+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:40.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:40.287+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:41.006 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:40.684+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:41.006 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:40.684+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:41.006 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:40.684+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:41.006 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:40.685+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:41.297 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:41.002+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:41.297 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:41.002+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:41.297 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:41.002+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:41.297 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:41.003+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:41.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:41.293+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:41.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:41.293+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:41.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:41.293+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:41.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:41.294+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:41.965 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:41.699+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:41.965 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:41.699+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:41.965 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:41.699+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:41.965 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:41.699+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:42.278 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:34:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:34:42.149+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 33218 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:34:42.278 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:41.961+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:42.279 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:41.961+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:42.279 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:41.961+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:42.279 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:41.962+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:42.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:42.302+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:42.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:42.302+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:42.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:42.302+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:42.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:42.303+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:43.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:42.947+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:43.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:42.947+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:43.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:42.947+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:43.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:42.948+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:43.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:42.746+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:43.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:42.746+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:43.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:42.746+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:43.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:42.747+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:43.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:43.296+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:43.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:43.296+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:43.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:43.296+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:43.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:43.297+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:44.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:43.904+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:44.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:43.904+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:44.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:43.904+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:44.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:43.905+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:44.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:43.735+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:44.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:43.735+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:44.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:43.735+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:44.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:43.736+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:44.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:44.316+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:44.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:44.316+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:44.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:44.316+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:44.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:44.317+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:44.925 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:34:44.926 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:34:45.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:44.873+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:45.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:44.874+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:45.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:44.874+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:45.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:44.874+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:45.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:44.713+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:45.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:44.713+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:45.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:44.713+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:45.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:44.713+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:45.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:45.306+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:45.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:45.306+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:45.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:45.306+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:45.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:45.307+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:46.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:45.904+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:46.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:45.904+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:46.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:45.904+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:46.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:45.905+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:46.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:45.727+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:46.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:45.727+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:46.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:45.727+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:46.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:45.727+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:46.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:46.346+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:46.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:46.346+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:46.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:46.346+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:46.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:46.347+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:47.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:46.914+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:47.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:46.914+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:47.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:46.914+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:47.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:46.915+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:47.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:46.687+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:47.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:46.687+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:47.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:46.687+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:47.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:46.688+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:47.407 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:34:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:34:47.153+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 33301 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:34:47.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:47.356+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:47.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:47.356+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:47.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:47.356+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:47.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:47.357+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:47.996 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:34:48.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:47.948+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:48.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:47.948+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:48.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:47.948+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:48.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:47.949+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:48.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:47.702+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:48.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:47.702+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:48.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:47.702+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:48.159 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:47.703+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:48.470 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:48.388+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:48.470 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:48.388+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:48.470 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:48.388+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:48.470 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:48.389+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:49.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:48.749+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:49.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:48.749+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:49.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:48.749+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:49.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:48.750+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:49.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:48.962+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:49.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:48.962+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:49.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:48.962+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:49.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:48.963+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:49.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:49.386+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:49.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:49.386+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:49.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:49.386+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:49.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:49.387+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:50.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:49.790+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:50.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:49.790+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:50.161 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:49.790+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:50.161 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:49.791+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:50.161 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:49.913+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:50.161 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:49.913+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:50.162 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:49.914+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:50.162 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:49.915+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:50.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:50.360+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:50.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:50.360+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:50.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:50.360+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:50.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:50.361+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:50.996 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:34:50.997 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:34:51.068 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:34:51.068 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-09T22:34:51.068 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-09T22:34:51.069 DEBUG:teuthology.orchestra.run.smithi006:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-02-09T22:34:51.102 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:50.788+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:51.102 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:50.788+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:51.102 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:50.788+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:51.103 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:50.789+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:51.103 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:50.935+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:51.103 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:50.935+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:51.103 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:50.935+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:51.103 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:50.935+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:51.108 INFO:teuthology.orchestra.run.smithi006.stdout:1707515805.992251 mon.smithi006 (mon.0) 690 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-02-09T22:34:51.109 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-02-09T22:34:51.109 INFO:tasks.cephadm:Compressing logs... 2024-02-09T22:34:51.109 DEBUG:teuthology.orchestra.run.smithi006:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-09T22:34:51.151 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:34:51.153 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:34:51.175 INFO:teuthology.orchestra.run.smithi006.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-02-09T22:34:51.177 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-09T22:34:51.177 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph-mgr.smithi006.glirxt.log 2024-02-09T22:34:51.177 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph.audit.log 2024-02-09T22:34:51.178 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph-osd.1.log 2024-02-09T22:34:51.178 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph-mgr.smithi006.glirxt.log: /var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph-mds.cephfs.smithi006.qyamnu.log 2024-02-09T22:34:51.178 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph-mon.smithi006.log 2024-02-09T22:34:51.179 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph-mds.cephfs.smithi006.qyamnu.log: gzip -5 --verbose -- /var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph-osd.0.log 2024-02-09T22:34:51.179 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph-mon.smithi006.log: 90.0% -- replaced with /var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph.audit.log.gz 2024-02-09T22:34:51.180 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph.cephadm.log 2024-02-09T22:34:51.184 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph-osd.0.log: 93.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-09T22:34:51.199 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph.log 2024-02-09T22:34:51.200 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph.cephadm.log: 81.5% -- replaced with /var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph.cephadm.log.gzgzip -5 --verbose -- /var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph-osd.2.log 2024-02-09T22:34:51.200 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-02-09T22:34:51.201 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph.log: 86.0% -- replaced with /var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph.log.gz 2024-02-09T22:34:51.206 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph-volume.log 2024-02-09T22:34:51.219 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph-mds.cephfs.smithi006.ypgiqs.log 2024-02-09T22:34:51.233 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph-volume.log: /var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph-mds.cephfs.smithi006.ypgiqs.log: 90.7% -- replaced with /var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph-volume.log.gz 2024-02-09T22:34:51.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:51.364+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:51.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:51.364+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:51.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:51.364+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:51.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:51.364+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:51.914 INFO:teuthology.orchestra.run.smithi006.stderr:gzip: /var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph-mgr.smithi006.glirxt.log: file size changed while zipping 2024-02-09T22:34:51.914 INFO:teuthology.orchestra.run.smithi006.stderr: 89.9% -- replaced with /var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph-mgr.smithi006.glirxt.log.gz 2024-02-09T22:34:52.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:51.775+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:52.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:51.775+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:52.160 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:51.775+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:52.161 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:51.776+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:52.161 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:51.944+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:52.161 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:51.944+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:52.161 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:51.944+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:52.161 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:51.945+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:52.164 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:34:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:34:52.157+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 33386 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:34:52.567 INFO:teuthology.orchestra.run.smithi006.stderr: 95.5% -- replaced with /var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph-mds.cephfs.smithi006.qyamnu.log.gz 2024-02-09T22:34:52.644 INFO:teuthology.orchestra.run.smithi006.stderr:gzip: /var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph-mon.smithi006.log: file size changed while zipping 2024-02-09T22:34:52.644 INFO:teuthology.orchestra.run.smithi006.stderr: 92.7% -- replaced with /var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph-mon.smithi006.log.gz 2024-02-09T22:34:52.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:52.360+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:52.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:52.360+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:52.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:52.360+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:52.659 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:52.361+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:53.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:52.966+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:53.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:52.966+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:53.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:52.966+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:53.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:52.967+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:53.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:52.781+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:53.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:52.781+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:53.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:52.781+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:53.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:52.782+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:53.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:53.327+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:53.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:53.327+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:53.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:53.327+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:53.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:53.328+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:54.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:53.789+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:54.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:53.790+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:54.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:53.790+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:54.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:53.791+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:54.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:53.987+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:54.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:53.987+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:54.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:53.987+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:54.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:53.988+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:54.204 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:34:54.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:54.296+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:54.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:54.296+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:54.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:54.296+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:54.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:54.297+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:55.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:54.751+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:55.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:54.751+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:55.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:54.751+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:55.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:54.752+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:55.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:54.971+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:55.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:54.971+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:55.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:54.971+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:55.159 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:54.973+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:55.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:55.299+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:55.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:55.299+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:55.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:55.299+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:55.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:55.300+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:56.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:55.780+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:56.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:55.780+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:56.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:55.780+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:56.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:55.781+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:56.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:55.979+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:56.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:55.979+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:56.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:55.979+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:56.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:55.980+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:56.234 INFO:teuthology.orchestra.run.smithi006.stderr:gzip: /var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph-osd.2.log: file size changed while zipping 2024-02-09T22:34:56.235 INFO:teuthology.orchestra.run.smithi006.stderr: 95.1% -- replaced with /var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph-osd.2.log.gz 2024-02-09T22:34:56.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:56.295+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:56.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:56.295+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:56.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:56.295+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:56.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:56.295+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:57.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:56.951+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:57.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:56.951+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:57.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:56.951+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:57.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:56.952+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:57.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:56.733+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:57.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:56.733+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:57.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:56.733+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:57.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:56.734+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:57.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:34:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:34:57.160+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 33469 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:34:57.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:57.339+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:57.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:57.339+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:57.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:57.339+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:57.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:57.341+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:58.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:57.996+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:58.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:57.996+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:58.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:57.996+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:58.158 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:57.997+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:58.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:57.764+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:58.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:57.764+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:58.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:57.764+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:58.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:57 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:57.765+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:58.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:58.338+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:58.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:58.338+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:58.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:58.338+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:58.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:58.339+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:59.018 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:58.724+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:59.018 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:58.724+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:59.019 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:58.724+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:59.019 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:58 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:58.724+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:34:59.346 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:59.014+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:59.347 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:59.014+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:59.347 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:59.014+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:34:59.347 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:34:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:34:59.015+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:59.601 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:59.343+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:59.601 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:59.343+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:59.601 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:59.343+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:34:59.601 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:34:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:34:59.344+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:34:59.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:59.758+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:59.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:59.758+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:59.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:59.758+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:34:59.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:34:59 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:34:59.759+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:00.224 INFO:teuthology.orchestra.run.smithi006.stderr:gzip: /var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph-osd.0.log: file size changed while zipping 2024-02-09T22:35:00.224 INFO:teuthology.orchestra.run.smithi006.stderr: 96.6% -- replaced with /var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph-osd.0.log.gz 2024-02-09T22:35:00.266 INFO:teuthology.orchestra.run.smithi006.stderr:gzip: /var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph-osd.1.log: file size changed while zipping 2024-02-09T22:35:00.266 INFO:teuthology.orchestra.run.smithi006.stderr: 96.5% -- replaced with /var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph-osd.1.log.gz 2024-02-09T22:35:00.374 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:00.027+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:00.375 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:00.027+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:00.375 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:00.027+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:00.375 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:00.028+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:00.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:00.369+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:00.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:00.369+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:00.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:00.369+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:00.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:00.370+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:01.064 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:00.721+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:01.065 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:00.721+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:01.065 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:00.721+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:01.065 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:00 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:00.721+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:01.395 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:01.061+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:01.396 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:01.061+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:01.396 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:01.061+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:01.396 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:01.062+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:01.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:01.392+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:01.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:01.392+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:01.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:01.392+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:01.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:01.393+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:02.099 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:01.700+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:02.099 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:01.700+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:02.099 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:01.700+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:02.099 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:01 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:01.701+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:02.380 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:35:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:35:02.165+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 33554 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:35:02.380 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:02.095+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:02.380 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:02.095+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:02.380 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:02.095+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:02.381 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:02.096+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:02.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:02.376+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:02.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:02.376+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:02.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:02.376+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:02.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:02.377+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:03.096 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:02.732+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:03.096 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:02.732+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:03.096 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:02.732+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:03.096 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:02 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:02.733+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:03.121 INFO:teuthology.orchestra.run.smithi006.stderr: 92.4% -- replaced with /var/log/ceph/ee26d820-c794-11ee-95b8-87774f69a715/ceph-mds.cephfs.smithi006.ypgiqs.log.gz 2024-02-09T22:35:03.123 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-02-09T22:35:03.123 INFO:teuthology.orchestra.run.smithi006.stderr:real 0m11.961s 2024-02-09T22:35:03.124 INFO:teuthology.orchestra.run.smithi006.stderr:user 0m37.653s 2024-02-09T22:35:03.124 INFO:teuthology.orchestra.run.smithi006.stderr:sys 0m0.839s 2024-02-09T22:35:03.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:35:03.206 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:35:03.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:03.092+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:03.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:03.092+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:03.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:03.092+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:03.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:03.093+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:03.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:03.329+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:03.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:03.329+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:03.411 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:03.329+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:03.411 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:03.330+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:03.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:03.694+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:03.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:03.694+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:03.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:03.694+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:03.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:03 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:03.695+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:04.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:04.123+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:04.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:04.123+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:04.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:04.123+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:04.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:04.124+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:04.340+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:04.340+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:04.340+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:04.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:04.341+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:05.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:04.693+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:05.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:04.693+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:05.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:04.693+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:05.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:04 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:04.694+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:05.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:05.174+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:05.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:05.174+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:05.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:05.174+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:05.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:05.175+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:05.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:05.319+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:05.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:05.319+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:05.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:05.319+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:05.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:05.319+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:06.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:05.718+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:06.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:05.718+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:06.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:05.718+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:06.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:05 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:05.719+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:06.268 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:35:06.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:06.213+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:06.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:06.213+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:06.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:06.213+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:06.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:06.214+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:06.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:06.283+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:06.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:06.283+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:06.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:06.283+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:06.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:06.284+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:07.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:06.725+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:07.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:06.725+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:07.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:06.725+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:07.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:06 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:06.726+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:07.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:35:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:35:07.169+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 33637 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:35:07.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:07.262+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:07.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:07.262+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:07.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:07.262+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:07.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:07.263+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:07.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:07.282+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:07.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:07.283+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:07.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:07.283+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:07.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:07.284+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:08.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:07.720+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:08.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:07.720+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:08.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:07.720+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:08.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:07 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:07.721+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:08.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:08.242+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:08.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:08.242+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:08.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:08.242+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:08.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:08.242+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:08.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:08.229+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:08.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:08.229+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:08.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:08.229+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:08.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:08.230+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:09.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:08.739+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:09.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:08.739+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:09.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:08.739+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:09.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:08 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:08.740+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:09.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:09.201+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:09.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:09.201+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:09.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:09.201+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:09.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:09.202+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:09.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:09.264+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:09.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:09.264+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:09.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:09.264+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:09.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:09.265+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:10.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:09.782+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:10.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:09.782+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:10.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:09.782+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:10.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:09 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:09.783+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:10.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:10.218+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:10.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:10.218+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:10.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:10.218+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:10.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:10.219+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:10.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:10.223+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:10.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:10.223+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:10.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:10.223+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:10.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:10.223+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:11.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:10.780+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:11.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:10.780+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:11.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:10.780+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:11.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:10 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:10.781+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:11.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:11.227+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:11.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:11.227+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:11.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:11.227+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:11.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:11.228+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:11.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:11.270+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:11.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:11.270+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:11.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:11.270+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:11.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:11.271+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:12.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:11.790+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:12.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:11.790+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:12.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:11.790+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:12.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:11 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:11.790+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:12.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:35:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:35:12.173+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 33722 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:35:12.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:12.271+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:12.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:12.271+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:12.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:12.271+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:12.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:12.272+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:12.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:12.188+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:12.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:12.188+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:12.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:12.188+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:12.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:12.189+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:13.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:12.791+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:13.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:12.791+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:13.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:12.791+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:13.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:12 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:12.792+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:13.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:13.294+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:13.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:13.294+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:13.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:13.294+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:13.499 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:13.295+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:13.499 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:13.214+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:13.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:13.214+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:13.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:13.214+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:13.500 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:13.215+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:14.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:13.761+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:14.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:13.761+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:14.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:13.761+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:14.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:13 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:13.762+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:14.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:14.187+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:14.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:14.187+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:14.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:14.187+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:14.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:14.188+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:14.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:14.326+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:14.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:14.326+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:14.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:14.326+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:14.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:14.326+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:15.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:14.771+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:15.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:14.771+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:15.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:14.771+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:15.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:14 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:14.772+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:15.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:15.207+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:15.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:15.207+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:15.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:15.207+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:15.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:15.208+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:15.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:15.374+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:15.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:15.374+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:15.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:15.374+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:15.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:15.375+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:16.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:15.758+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:16.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:15.758+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:16.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:15.758+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:16.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:15 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:15.759+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:16.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:16.253+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:16.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:16.253+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:16.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:16.253+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:16.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:16.254+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:16.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:16.394+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:16.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:16.394+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:16.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:16.394+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:16.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:16.395+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:17.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:16.740+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:17.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:16.740+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:17.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:16.740+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:17.158 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:16 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:16.740+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:17.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:35:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:35:17.177+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 33805 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:35:17.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:17.395+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:17.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:17.395+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:17.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:17.395+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:17.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:17.396+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:17.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:17.236+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:17.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:17.236+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:17.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:17.236+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:17.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:17.237+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:18.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:17.755+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:18.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:17.755+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:18.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:17.755+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:18.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:17 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:17.756+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:18.269 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:35:18.270 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:35:18.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:18.369+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:18.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:18.369+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:18.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:18.369+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:18.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:18.370+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:18.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:18.278+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:18.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:18.278+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:18.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:18.278+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:18.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:18.280+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:19.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:18.709+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:19.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:18.709+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:19.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:18.709+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:19.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:18 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:18.710+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:19.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:19.329+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:19.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:19.329+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:19.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:19.329+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:19.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:19.330+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:19.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:19.353+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:19.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:19.353+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:19.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:19.353+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:19.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:19.354+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:20.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:19.691+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:20.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:19.691+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:20.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:19.691+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:20.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:19 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:19.691+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:20.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:20.308+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:20.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:20.308+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:20.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:20.308+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:20.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:20.309+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:20.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:20.372+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:20.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:20.372+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:20.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:20.372+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:20.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:20.373+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:21.037 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:20.662+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:21.037 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:20.662+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:21.038 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:20.662+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:21.038 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:20 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:20.663+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:21.344 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:35:21.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:21.306+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:21.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:21.306+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:21.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:21.306+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:21.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:21.307+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:21.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:21.398+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:21.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:21.398+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:21.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:21.398+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:21.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:21.399+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:22.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:21.684+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:22.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:21.684+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:22.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:21.684+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:22.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:21 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:21.684+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:22.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:35:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:35:22.181+0000 7fa7d707e700 -1 mon.smithi006@0(probing) e2 get_health_metrics reporting 33890 slow ops, oldest is log(1 entries from seq 275 at 2024-02-09T21:58:24.891898+0000) 2024-02-09T22:35:22.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:22.360+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:22.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:22.360+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:22.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:22.360+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:22.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:22.361+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:22.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:22.353+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:22.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:22.353+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:22.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:22.353+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:22.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:22.354+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:23.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:22.667+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:23.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:22.667+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:23.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:22.667+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:23.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:22 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:22.668+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:23.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:23.331+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:23.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:23.331+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:23.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:23.331+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:23.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:23.332+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:23.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:23.332+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:23.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:23.332+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:23.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:23.332+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:23.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:23.332+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:24.109 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:23.669+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:24.110 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:23.669+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:24.110 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:23.669+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:24.110 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:23 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:23.669+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:24.345 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:35:24.346 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:35:24.384 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:24.290+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:24.384 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:24.290+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:24.385 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:24.290+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:24.385 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:24.291+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:24.416 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:35:24.416 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-09T22:35:24.416 INFO:tasks.cephadm:Removing cluster... 2024-02-09T22:35:24.417 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ee26d820-c794-11ee-95b8-87774f69a715 --force 2024-02-09T22:35:24.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:24.627+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:24.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:24.627+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:24.657 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:24.627+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:24.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:24.627+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:24.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:24.379+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:24.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:24.379+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:24.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:24.379+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:24.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:24 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:24.380+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:25.016 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:35:24 smithi006 systemd[1]: Stopping Ceph mon.smithi006 for ee26d820-c794-11ee-95b8-87774f69a715... 2024-02-09T22:35:25.381 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:35:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:35:25.011+0000 7fa7da084700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi006 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T22:35:25.382 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:35:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006[29254]: 2024-02-09T22:35:25.011+0000 7fa7da084700 -1 mon.smithi006@0(probing) e2 *** Got Signal Terminated *** 2024-02-09T22:35:25.382 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:25.251+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:25.382 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:25.251+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:25.382 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:25.251+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:25.382 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:25.251+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:25.657 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:35:25 smithi006 podman[85877]: 2024-02-09 22:35:25.378956947 +0000 UTC m=+0.493738180 container died f8cba7495eab196f0f556df5598c0e7b667a146011c07d6ae2ff27f1a7b78f0d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=quincy-72ce7c1, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS) 2024-02-09T22:35:25.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:25.402+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:25.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:25.402+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:25.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:25.402+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:25.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:25.403+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:25.917 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:25.656+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:25.917 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:25.656+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:25.917 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:25.656+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:25.918 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:25 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:25.657+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:26.264 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:35:25 smithi006 podman[85877]: 2024-02-09 22:35:25.914645106 +0000 UTC m=+1.029426332 container cleanup f8cba7495eab196f0f556df5598c0e7b667a146011c07d6ae2ff27f1a7b78f0d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006, CEPH_POINT_RELEASE=, RELEASE=quincy-72ce7c1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-09T22:35:26.264 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:35:25 smithi006 bash[85877]: ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006 2024-02-09T22:35:26.264 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:35:26 smithi006 podman[85891]: 2024-02-09 22:35:26.149695572 +0000 UTC m=+0.773692222 container remove f8cba7495eab196f0f556df5598c0e7b667a146011c07d6ae2ff27f1a7b78f0d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-mon-smithi006, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=quincy-72ce7c1) 2024-02-09T22:35:26.617 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:35:26 smithi006 systemd[1]: ceph-ee26d820-c794-11ee-95b8-87774f69a715@mon.smithi006.service: Deactivated successfully. 2024-02-09T22:35:26.618 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:35:26 smithi006 systemd[1]: Stopped Ceph mon.smithi006 for ee26d820-c794-11ee-95b8-87774f69a715. 2024-02-09T22:35:26.618 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 09 22:35:26 smithi006 systemd[1]: ceph-ee26d820-c794-11ee-95b8-87774f69a715@mon.smithi006.service: Consumed 23.539s CPU time. 2024-02-09T22:35:26.618 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:26.402+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:26.618 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:26.402+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:26.618 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:26.402+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:26.619 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:26.403+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:26.619 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:26.259+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:26.619 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:26.259+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:26.619 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:26.259+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:26.619 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:26.260+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:26.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:26.612+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:26.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:26.612+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:26.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:26.612+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:26.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:26 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:26.613+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:27.592 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:27.363+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:27.592 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:27.363+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:27.593 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:27.363+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:27.593 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:27.364+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:27.593 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:27.211+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:27.593 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:27.211+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:27.593 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:27.211+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:27.593 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:27.212+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:27.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:27.635+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:27.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:27.635+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:27.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:27.635+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:27.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:27 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:27.636+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:28.539 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:28.216+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:28.539 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:28.216+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:28.540 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:28.216+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:28.540 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:28.217+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:28.540 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:28.412+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:28.540 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:28.412+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:28.540 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:28.412+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:28.540 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:28.412+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:28.906 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:28.632+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:28.906 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:28.632+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:28.906 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:28.632+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:28.906 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:28 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:28.632+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:29.460 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:29.200+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:29.461 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:29.200+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:29.461 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:29.200+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:29.461 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:29.201+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:29.836 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:29.456+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:29.836 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:29.456+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:29.837 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:29.456+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:29.837 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:29.456+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:29.837 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:29.631+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:29.837 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:29.631+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:29.837 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:29.631+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:29.837 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:29 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:29.631+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:30.591 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:30.230+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:30.591 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:30.230+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:30.592 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:30.230+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:30.592 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:30.231+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:30.592 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:30.438+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:30.592 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:30.438+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:30.592 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:30.438+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:30.592 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:30.439+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:30.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:30.678+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:30.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:30.678+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:30.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:30.678+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:30.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:30 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:30.678+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:31.593 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:31.424+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:31.593 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:31.424+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:31.593 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:31.424+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:31.593 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:31.425+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:31.594 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:31.269+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:31.594 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:31.269+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:31.594 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:31.269+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:31.594 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:31.270+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:31.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:31.709+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:31.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:31.709+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:31.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:31.709+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:31.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:31 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:31.710+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:32.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:32.397+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:32.657 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:32.397+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:32.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:32.397+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:32.658 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:32.398+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:32.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:32.310+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:32.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:32.310+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:32.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:32.310+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:32.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:32.311+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:33.149 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:32.687+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:33.150 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:32.687+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:33.150 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:32.687+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:33.150 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:32 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:32.688+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:33.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:33.327+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:33.407 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:33.327+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:33.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:33.327+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:33.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:33.328+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:33.778 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:33.409+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:33.779 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:33.409+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:33.779 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:33.409+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:33.779 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:33.410+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:33.779 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:33.657+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:33.779 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:33.657+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:33.780 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:33.657+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:33.780 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:33 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:33.658+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:34.547 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:34.404+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:34.547 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:34.404+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:34.547 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:34.404+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:34.547 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:34.405+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:34.547 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:34.321+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:34.547 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:34.321+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:34.548 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:34.321+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:34.548 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:34.322+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:34.799 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:34.640+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:34.800 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:34.640+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:34.800 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:34.640+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:34.800 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:34 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:34.641+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:35.157 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:35 smithi006 systemd[1]: Stopping Ceph osd.0 for ee26d820-c794-11ee-95b8-87774f69a715... 2024-02-09T22:35:35.621 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:35.317+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:35.621 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:35.317+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:35.621 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:35.317+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:35.621 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:35.318+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:35.622 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:35.297+0000 7f5b13bcb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T22:35:35.622 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:35.297+0000 7f5b13bcb700 -1 osd.0 45 *** Got signal Terminated *** 2024-02-09T22:35:35.622 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:35.297+0000 7f5b13bcb700 -1 osd.0 45 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-09T22:35:35.622 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:35.430+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:35.622 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:35.430+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:35.622 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:35.430+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:35.622 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:35.431+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:35.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:35.617+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:35.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:35.617+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:35.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:35.617+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:35.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:35 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:35.618+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:36.613 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:36.384+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:36.613 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:36.384+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:36.614 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:36.384+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:36.614 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:36.385+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:36.614 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:36.274+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:36.614 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:36.274+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:36.614 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:36.274+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:36.614 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:36.275+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:36.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:36.609+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:36.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:36.609+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:36.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:36.609+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:36.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:36 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:36.610+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:37.615 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:37.427+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:37.615 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:37.427+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:37.616 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:37.427+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:37.616 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:37.427+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:37.616 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:37.306+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:37.616 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:37.306+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:37.616 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:37.306+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:37.616 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:37.307+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:37.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:37.611+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:37.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:37.611+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:37.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:37.611+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:37.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:37 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:37.612+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:38.570 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:38.285+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:38.571 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:38.285+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:38.571 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:38.285+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:38.571 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:38.285+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:38.571 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:38.390+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:38.571 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:38.390+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:38.572 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:38.390+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:38.572 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:38.391+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:38.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:38.567+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:38.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:38.567+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:38.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:38.567+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:38.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:38 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:38.568+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:39.580 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:39.405+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:39.580 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:39.405+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:39.580 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:39.405+0000 7f5b0d1c3700 -1 osd.0 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:08.640498+0000 (oldest deadline 2024-02-09T22:30:28.640498+0000) 2024-02-09T22:35:39.580 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0[48305]: 2024-02-09T22:35:39.406+0000 7f5b0d1c3700 -1 osd.0 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:266 2.13 2:ceffbd45:::200.0000000a:head [write 724800~7976 in=7976b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:39.581 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:39.248+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:39.581 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:39.248+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:39.581 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:39.248+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:39.581 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:39.249+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:39.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:39.576+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:39.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:39.576+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:39.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:39.576+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:39.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:39 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:39.577+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:40.604 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:40.298+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:40.605 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:40.298+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:40.605 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:40.298+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:40.605 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:40.299+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:40.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:40 smithi006 podman[86707]: 2024-02-09 22:35:40.745754416 +0000 UTC m=+5.574229259 container died 1aea34e55f656062b5b54636f7b46f523ac2f142db714af0a15ad36b2bdbfe80 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=quincy-72ce7c1, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-02-09T22:35:40.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:40.601+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:40.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:40.601+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:40.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:40.601+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:40.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:40 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:40.602+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:41.315 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:41 smithi006 podman[86707]: 2024-02-09 22:35:41.131825305 +0000 UTC m=+5.960300153 container cleanup 1aea34e55f656062b5b54636f7b46f523ac2f142db714af0a15ad36b2bdbfe80 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=quincy-72ce7c1, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-09T22:35:41.315 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:41 smithi006 bash[86707]: ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0 2024-02-09T22:35:41.580 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:41 smithi006 podman[86723]: 2024-02-09 22:35:41.391820588 +0000 UTC m=+0.644307255 container remove 1aea34e55f656062b5b54636f7b46f523ac2f142db714af0a15ad36b2bdbfe80 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-72ce7c1, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-02-09T22:35:41.580 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:41.311+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:41.581 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:41.311+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:41.581 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:41.311+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:41.581 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:41.312+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:41.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:41 smithi006 podman[86812]: 2024-02-09 22:35:41.68707822 +0000 UTC m=+0.018544863 image pull a2a0bd126c96301261ac6c5708253ef5b41bd73b23624c6dbd2eed86fc57c9b5 quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc 2024-02-09T22:35:41.907 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:41 smithi006 podman[86812]: 2024-02-09 22:35:41.809679935 +0000 UTC m=+0.141146577 container create 4edfca478e147072dfeb4ceda9baa643423746c9c9830e8080d10f33991b6fb9 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_CLEAN=True, RELEASE=quincy-72ce7c1, GIT_BRANCH=HEAD) 2024-02-09T22:35:41.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:41.576+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:41.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:41.576+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:41.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:41.576+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:41.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:41 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:41.577+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:42.281 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:42 smithi006 podman[86812]: 2024-02-09 22:35:42.070719776 +0000 UTC m=+0.402186419 container init 4edfca478e147072dfeb4ceda9baa643423746c9c9830e8080d10f33991b6fb9 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , RELEASE=quincy-72ce7c1, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-09T22:35:42.281 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:42 smithi006 podman[86812]: 2024-02-09 22:35:42.07385342 +0000 UTC m=+0.405320066 container start 4edfca478e147072dfeb4ceda9baa643423746c9c9830e8080d10f33991b6fb9 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=quincy-72ce7c1, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131) 2024-02-09T22:35:42.281 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:42 smithi006 podman[86812]: 2024-02-09 22:35:42.162543372 +0000 UTC m=+0.494010024 container attach 4edfca478e147072dfeb4ceda9baa643423746c9c9830e8080d10f33991b6fb9 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=quincy-72ce7c1, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-02-09T22:35:42.558 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:42 smithi006 podman[86812]: 2024-02-09 22:35:42.385140332 +0000 UTC m=+0.716606977 container died 4edfca478e147072dfeb4ceda9baa643423746c9c9830e8080d10f33991b6fb9 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=quincy-72ce7c1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-09T22:35:42.558 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:42.277+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:42.559 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:42.277+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:42.559 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:42.277+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:42.559 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:42.278+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:42.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:42.565+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:42.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:42.565+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:42.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:42.565+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:42.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:42 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:42.566+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:43.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:43 smithi006 podman[86839]: 2024-02-09 22:35:43.076985992 +0000 UTC m=+0.682686216 container remove 4edfca478e147072dfeb4ceda9baa643423746c9c9830e8080d10f33991b6fb9 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-72ce7c1) 2024-02-09T22:35:43.407 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:43 smithi006 systemd[1]: ceph-ee26d820-c794-11ee-95b8-87774f69a715@osd.0.service: Deactivated successfully. 2024-02-09T22:35:43.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:43 smithi006 systemd[1]: Stopped Ceph osd.0 for ee26d820-c794-11ee-95b8-87774f69a715. 2024-02-09T22:35:43.408 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 09 22:35:43 smithi006 systemd[1]: ceph-ee26d820-c794-11ee-95b8-87774f69a715@osd.0.service: Consumed 35min 47.503s CPU time. 2024-02-09T22:35:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:43.259+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:43.259+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:43.259+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:43.408 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:43.260+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:43.409 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:43 smithi006 systemd[1]: Stopping Ceph osd.1 for ee26d820-c794-11ee-95b8-87774f69a715... 2024-02-09T22:35:43.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:43.587+0000 7f83492c2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T22:35:43.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:43.587+0000 7f83492c2700 -1 osd.1 45 *** Got signal Terminated *** 2024-02-09T22:35:43.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:43.587+0000 7f83492c2700 -1 osd.1 45 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-09T22:35:43.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:43.580+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:43.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:43.580+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:43.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:43.580+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:43.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:43 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:43.581+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:44.568 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:44.253+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:44.569 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:44.253+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:44.569 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:44.254+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:44.569 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:44.255+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:44.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:44.565+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:44.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:44.565+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:44.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:44.565+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:44.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:44 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:44.566+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:45.537 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:45.281+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:45.537 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:45.281+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:45.537 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:45.281+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:45.537 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:45.282+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:45.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:45.533+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:45.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:45.533+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:45.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:45.533+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:45.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:45 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:45.534+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:46.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:46.274+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:46.657 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:46.274+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:46.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:46.274+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:46.658 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:46.275+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:46.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:46.489+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:46.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:46.489+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:46.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:46.489+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:46.658 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:46 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:46.489+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:47.558 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:47.273+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:47.559 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:47.273+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:47.559 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:47.273+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:47.559 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:47.273+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:47.559 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:47.464+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:47.559 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:47.464+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:47.559 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:47.464+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:47.560 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:47 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:47.465+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:48.567 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:48.275+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:48.568 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:48.275+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:48.568 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:48.275+0000 7f83448be700 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:30:19.425317+0000 (oldest deadline 2024-02-09T22:30:39.425317+0000) 2024-02-09T22:35:48.568 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1[56402]: 2024-02-09T22:35:48.276+0000 7f83448be700 -1 osd.1 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.4:265 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=66b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45) 2024-02-09T22:35:48.568 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:48.484+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:48.568 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:48.484+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:48.569 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:48.484+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:48.569 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:48 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:48.485+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:49.492 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:49 smithi006 podman[86924]: 2024-02-09 22:35:49.200157556 +0000 UTC m=+5.745028682 container died 58f54ee70423b37d22a458e4b62423a0d87b4b9a9764680308176f542b79897f (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1, RELEASE=quincy-72ce7c1, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-09T22:35:49.809 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:49 smithi006 podman[86924]: 2024-02-09 22:35:49.579936717 +0000 UTC m=+6.124807851 container cleanup 58f54ee70423b37d22a458e4b62423a0d87b4b9a9764680308176f542b79897f (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , RELEASE=quincy-72ce7c1, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-02-09T22:35:49.809 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:49 smithi006 bash[86924]: ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1 2024-02-09T22:35:49.809 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:49.523+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:49.810 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:49.523+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:49.810 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:49.523+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:49.810 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:49 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:49.524+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:50.157 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:49 smithi006 podman[86942]: 2024-02-09 22:35:49.806539378 +0000 UTC m=+0.603376727 container remove 58f54ee70423b37d22a458e4b62423a0d87b4b9a9764680308176f542b79897f (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=quincy-72ce7c1, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-02-09T22:35:50.504 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:50 smithi006 podman[87032]: 2024-02-09 22:35:50.150436954 +0000 UTC m=+0.017576208 image pull a2a0bd126c96301261ac6c5708253ef5b41bd73b23624c6dbd2eed86fc57c9b5 quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc 2024-02-09T22:35:50.504 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:50 smithi006 podman[87032]: 2024-02-09 22:35:50.2658179 +0000 UTC m=+0.132957144 container create 31fa3d817fd1dae8f4445b02275a2b1c3966595de6b0cf59151f9609ca789526 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-72ce7c1, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True) 2024-02-09T22:35:50.791 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:50.517+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:50.791 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:50.517+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:50.791 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:50.517+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:50.792 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:50 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:50.518+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:50.792 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:50 smithi006 podman[87032]: 2024-02-09 22:35:50.501726385 +0000 UTC m=+0.368865645 container init 31fa3d817fd1dae8f4445b02275a2b1c3966595de6b0cf59151f9609ca789526 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=quincy-72ce7c1, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True) 2024-02-09T22:35:50.792 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:50 smithi006 podman[87032]: 2024-02-09 22:35:50.504803856 +0000 UTC m=+0.371943100 container start 31fa3d817fd1dae8f4445b02275a2b1c3966595de6b0cf59151f9609ca789526 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1-deactivate, RELEASE=quincy-72ce7c1, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-02-09T22:35:50.792 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:50 smithi006 podman[87032]: 2024-02-09 22:35:50.601828321 +0000 UTC m=+0.468967580 container attach 31fa3d817fd1dae8f4445b02275a2b1c3966595de6b0cf59151f9609ca789526 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.2, RELEASE=quincy-72ce7c1, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-02-09T22:35:51.141 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:50 smithi006 podman[87032]: 2024-02-09 22:35:50.788604493 +0000 UTC m=+0.655743748 container died 31fa3d817fd1dae8f4445b02275a2b1c3966595de6b0cf59151f9609ca789526 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, maintainer=Guillaume Abrioux , RELEASE=quincy-72ce7c1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-09T22:35:51.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:51 smithi006 podman[87058]: 2024-02-09 22:35:51.472700382 +0000 UTC m=+0.678639019 container remove 31fa3d817fd1dae8f4445b02275a2b1c3966595de6b0cf59151f9609ca789526 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=quincy-72ce7c1, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-02-09T22:35:51.907 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:51 smithi006 systemd[1]: ceph-ee26d820-c794-11ee-95b8-87774f69a715@osd.1.service: Deactivated successfully. 2024-02-09T22:35:51.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:51 smithi006 systemd[1]: Stopped Ceph osd.1 for ee26d820-c794-11ee-95b8-87774f69a715. 2024-02-09T22:35:51.908 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 09 22:35:51 smithi006 systemd[1]: ceph-ee26d820-c794-11ee-95b8-87774f69a715@osd.1.service: Consumed 35min 55.383s CPU time. 2024-02-09T22:35:51.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:51.492+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:51.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:51.492+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:51.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:51.492+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:51.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:51.493+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:51.909 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:51 smithi006 systemd[1]: Stopping Ceph osd.2 for ee26d820-c794-11ee-95b8-87774f69a715... 2024-02-09T22:35:52.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:51.959+0000 7f93c7a0a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T22:35:52.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:51.959+0000 7f93c7a0a700 -1 osd.2 45 *** Got signal Terminated *** 2024-02-09T22:35:52.407 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:51 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:51.959+0000 7f93c7a0a700 -1 osd.2 45 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-09T22:35:52.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:52.461+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:52.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:52.461+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:52.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:52.461+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:52.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:52 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:52.462+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:53.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:53.466+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:53.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:53.466+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:53.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:53.466+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:53.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:53 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:53.467+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:54.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:54.483+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:54.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:54.483+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:54.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:54.483+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:54.908 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:54 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:54.484+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:55.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:55.489+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:55.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:55.489+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:55.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:55.489+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:55.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:55 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:55.490+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:56.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:56.521+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6804 osd.3 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:56.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:56.521+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6812 osd.4 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:56.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:56.521+0000 7f93c580b700 -1 osd.2 45 heartbeat_check: no reply from 172.21.15.90:6820 osd.5 ever on either front or back, first ping sent 2024-02-09T22:29:57.987474+0000 (oldest deadline 2024-02-09T22:30:17.987474+0000) 2024-02-09T22:35:56.907 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:56 smithi006 ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2[63757]: 2024-02-09T22:35:56.521+0000 7f93c580b700 -1 osd.2 45 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:1876 3.1e 3:78ab6fe9:::200000001e8.00000000:head [write 0~8142 in=8142b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-02-09T22:35:57.663 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:57 smithi006 podman[87137]: 2024-02-09 22:35:57.406384232 +0000 UTC m=+5.575489818 container died 9dfe76e205a62409db2e671480027eaf9309ef8096a0e448f80bf5c40ba410d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2, maintainer=Guillaume Abrioux , RELEASE=quincy-72ce7c1, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2) 2024-02-09T22:35:58.022 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:57 smithi006 podman[87137]: 2024-02-09 22:35:57.767910879 +0000 UTC m=+5.937016454 container cleanup 9dfe76e205a62409db2e671480027eaf9309ef8096a0e448f80bf5c40ba410d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2, ceph=True, maintainer=Guillaume Abrioux , RELEASE=quincy-72ce7c1, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-09T22:35:58.022 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:57 smithi006 bash[87137]: ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2 2024-02-09T22:35:58.336 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:58 smithi006 podman[87155]: 2024-02-09 22:35:58.01925963 +0000 UTC m=+0.612055034 container remove 9dfe76e205a62409db2e671480027eaf9309ef8096a0e448f80bf5c40ba410d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, RELEASE=quincy-72ce7c1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-02-09T22:35:58.608 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:58 smithi006 podman[87247]: 2024-02-09 22:35:58.236705973 +0000 UTC m=+0.017402407 image pull a2a0bd126c96301261ac6c5708253ef5b41bd73b23624c6dbd2eed86fc57c9b5 quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc 2024-02-09T22:35:58.608 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:58 smithi006 podman[87247]: 2024-02-09 22:35:58.345069173 +0000 UTC m=+0.125765634 container create 5268c2cafff0039574a24f7123c606b80d3d52201e6c12f0ea3609845bd55698 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=quincy-72ce7c1, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-09T22:35:58.608 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:58 smithi006 podman[87247]: 2024-02-09 22:35:58.572652124 +0000 UTC m=+0.353348542 container init 5268c2cafff0039574a24f7123c606b80d3d52201e6c12f0ea3609845bd55698 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, RELEASE=quincy-72ce7c1, io.buildah.version=1.33.2) 2024-02-09T22:35:58.608 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:58 smithi006 podman[87247]: 2024-02-09 22:35:58.576599408 +0000 UTC m=+0.357295835 container start 5268c2cafff0039574a24f7123c606b80d3d52201e6c12f0ea3609845bd55698 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=quincy-72ce7c1, io.buildah.version=1.33.2) 2024-02-09T22:35:58.878 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:58 smithi006 podman[87247]: 2024-02-09 22:35:58.6644518 +0000 UTC m=+0.445148233 container attach 5268c2cafff0039574a24f7123c606b80d3d52201e6c12f0ea3609845bd55698 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , RELEASE=quincy-72ce7c1) 2024-02-09T22:35:59.157 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:58 smithi006 podman[87247]: 2024-02-09 22:35:58.876260726 +0000 UTC m=+0.656957145 container died 5268c2cafff0039574a24f7123c606b80d3d52201e6c12f0ea3609845bd55698 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=quincy-72ce7c1, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-02-09T22:35:59.426 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:59 smithi006 podman[87273]: 2024-02-09 22:35:59.409310255 +0000 UTC m=+0.527864082 container remove 5268c2cafff0039574a24f7123c606b80d3d52201e6c12f0ea3609845bd55698 (image=quay.ceph.io/ceph-ci/ceph@sha256:5984db8535021d0560ce9082f3917f2ac2d4b60780607d50d04c248b8beb85dc, name=ceph-ee26d820-c794-11ee-95b8-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=quincy-72ce7c1, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131) 2024-02-09T22:35:59.875 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:59 smithi006 systemd[1]: ceph-ee26d820-c794-11ee-95b8-87774f69a715@osd.2.service: Deactivated successfully. 2024-02-09T22:35:59.875 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:59 smithi006 systemd[1]: Stopped Ceph osd.2 for ee26d820-c794-11ee-95b8-87774f69a715. 2024-02-09T22:35:59.875 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 09 22:35:59 smithi006 systemd[1]: ceph-ee26d820-c794-11ee-95b8-87774f69a715@osd.2.service: Consumed 18min 19.862s CPU time. 2024-02-09T22:36:02.476 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:36:02.477 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:36:05.532 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:36:14.533 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:36:14.534 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:36:17.596 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:36:29.596 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:36:29.597 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:36:32.668 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:36:35.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:36:35.670 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:36:35.740 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:36:35.740 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-09T22:36:35.740 INFO:tasks.cephadm:Teardown complete 2024-02-09T22:36:35.741 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 1591, in distribute_config_and_admin_keyring yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 1845, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 1848, in task log.info('Teardown begin') File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 1724, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 180, in download_cephadm _rm_cluster(ctx, cluster_name) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephadm.py", line 348, in _rm_cluster ctx.cluster.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi090 2024-02-09T22:36:35.761 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-09T22:36:35.772 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-02-09T22:36:35.787 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-09T22:36:35.787 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-09T22:36:35.803 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:36:35.803 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:36:38.876 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:36:47.877 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:36:47.878 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:36:50.940 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:37:02.941 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:37:02.943 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:37:06.012 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:37:09.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:37:09.013 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:37:09.084 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:37:09.084 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-09T22:37:09.128 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi090 2024-02-09T22:37:09.147 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-09T22:37:09.159 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-09T22:37:09.159 DEBUG:teuthology.orchestra.run.smithi006:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-09T22:37:09.166 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:37:09.167 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:37:09.181 INFO:teuthology.orchestra.run.smithi006.stderr:bash: line 1: ntpq: command not found 2024-02-09T22:37:09.186 INFO:teuthology.orchestra.run.smithi006.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-09T22:37:09.186 INFO:teuthology.orchestra.run.smithi006.stdout:=============================================================================== 2024-02-09T22:37:09.186 INFO:teuthology.orchestra.run.smithi006.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 29 -111us[ -132us] +/- 89ms 2024-02-09T22:37:09.186 INFO:teuthology.orchestra.run.smithi006.stdout:^* hv02.front.sepia.ceph.com 3 7 377 25 -1416us[-1437us] +/- 55ms 2024-02-09T22:37:09.186 INFO:teuthology.orchestra.run.smithi006.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 25 +2688us[+2667us] +/- 96ms 2024-02-09T22:37:09.187 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T22:37:12.220 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:37:21.221 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:37:21.222 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:37:24.288 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:37:36.289 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:37:36.290 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:37:39.360 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:37:42.361 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:37:42.362 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:37:42.428 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:37:42.428 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-09T22:37:42.429 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi090 2024-02-09T22:37:42.443 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-09T22:37:42.456 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-09T22:37:42.456 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-09T22:37:42.503 DEBUG:teuthology.orchestra.run.smithi006:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-09T22:37:42.591 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:37:42.592 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:37:45.660 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:37:54.661 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:37:54.662 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:37:57.728 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:38:09.729 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:38:09.730 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:38:12.796 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:38:15.797 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:38:15.798 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:38:15.872 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:38:15.878 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-09T22:38:15.878 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi090 2024-02-09T22:38:15.886 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-09T22:38:15.922 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-09T22:38:15.954 INFO:teuthology.task.internal:Duration was 3397.932916 seconds 2024-02-09T22:38:15.954 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-09T22:38:15.971 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-09T22:38:15.971 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-09T22:38:15.977 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:38:15.977 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:38:16.024 INFO:teuthology.orchestra.run.smithi006.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-09T22:38:19.040 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:38:28.041 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:38:28.042 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:38:31.104 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:38:43.105 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:38:43.106 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:38:46.172 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:38:49.173 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:38:49.174 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:38:49.244 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:38:49.244 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-09T22:38:49.245 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi090 2024-02-09T22:38:49.262 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-09T22:38:49.276 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-09T22:38:49.276 DEBUG:teuthology.orchestra.run.smithi006:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-09T22:38:49.313 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:38:49.313 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:38:52.380 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:39:01.380 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:39:01.381 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:39:04.448 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:39:16.449 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:39:16.450 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:39:19.516 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:39:22.517 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:39:22.518 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:39:22.588 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:39:22.588 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-09T22:39:22.589 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi090 2024-02-09T22:39:22.605 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-09T22:39:22.620 DEBUG:teuthology.orchestra.run.smithi006:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-09T22:39:22.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:39:22.627 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:39:22.654 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = core 2024-02-09T22:39:25.692 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:39:34.693 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:39:34.694 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:39:37.756 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:39:49.757 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:39:49.758 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:39:52.832 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:39:55.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:39:55.834 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:39:55.903 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:39:55.903 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-09T22:39:55.904 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi090 2024-02-09T22:39:55.921 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-09T22:39:55.935 INFO:teuthology.task.internal:Transferring archived files... 2024-02-09T22:39:55.936 DEBUG:teuthology.misc:Transferring archived files from smithi006:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi/7554325/remote/smithi006 2024-02-09T22:39:55.937 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-09T22:39:55.979 DEBUG:teuthology.misc:Transferring archived files from smithi090:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi/7554325/remote/smithi090 2024-02-09T22:39:55.979 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:39:55.980 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:39:59.036 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:40:08.037 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:40:08.039 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:40:11.104 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:40:23.105 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:40:23.106 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:40:26.172 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:40:29.173 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:40:29.174 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:40:29.248 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:40:29.248 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-09T22:40:29.249 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi090 2024-02-09T22:40:29.266 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-09T22:40:29.278 INFO:teuthology.task.internal:Not uploading archives. 2024-02-09T22:40:29.279 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-09T22:40:29.298 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-09T22:40:29.298 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-09T22:40:29.306 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:40:29.306 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:40:29.321 INFO:teuthology.orchestra.run.smithi006.stdout: 262166 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 9 22:36 /home/ubuntu/cephtest 2024-02-09T22:40:29.321 INFO:teuthology.orchestra.run.smithi006.stdout: 262167 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Feb 9 22:39 /home/ubuntu/cephtest/archive 2024-02-09T22:40:29.321 INFO:teuthology.orchestra.run.smithi006.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 9 21:41 /home/ubuntu/cephtest/archive/syslog 2024-02-09T22:40:29.321 INFO:teuthology.orchestra.run.smithi006.stdout: 265294 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Feb 9 21:41 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-09T22:40:29.321 INFO:teuthology.orchestra.run.smithi006.stdout: 265293 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Feb 9 21:41 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-09T22:40:29.322 INFO:teuthology.orchestra.run.smithi006.stdout: 402361 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 9 22:37 /home/ubuntu/cephtest/archive/audit 2024-02-09T22:40:29.322 INFO:teuthology.orchestra.run.smithi006.stdout: 426294 176 -rw------- 1 ubuntu ubuntu 176663 Feb 9 22:37 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-02-09T22:40:29.322 INFO:teuthology.orchestra.run.smithi006.stdout: 265373 360 -rwxr-xr-x 1 ubuntu ubuntu 367151 Feb 9 21:48 /home/ubuntu/cephtest/cephadm 2024-02-09T22:40:29.323 INFO:teuthology.orchestra.run.smithi006.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-02-09T22:40:32.380 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:40:41.381 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:40:41.382 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:40:44.444 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:40:56.445 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:40:56.446 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:40:59.516 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:41:02.517 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-09T22:41:02.518 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:41:02.588 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-09T22:41:02.588 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-09T22:41:02.589 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi090 2024-02-09T22:41:02.605 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-09T22:41:02.617 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-09T22:41:02.656 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: OSError: [Errno 113] No route to host 2024-02-09T22:41:02.656 INFO:teuthology.nuke:Checking targets against current locks 2024-02-09T22:41:02.685 DEBUG:teuthology.nuke:shortname: smithi006 2024-02-09T22:41:02.686 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-09T22:41:02.711 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi/7554325', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-09 21:25:11.194102', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNL5Lq/5A0kvQ8OuALLL7kAYuBa8yO7d4VjvaOmaK7KrVmb52FFr2ssHWu7VhEHtmrdm/qgHO3FAWCgEk8BaI5I='} 2024-02-09T22:41:02.716 DEBUG:teuthology.nuke:shortname: smithi090 2024-02-09T22:41:02.716 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-09T22:41:02.740 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi/7554325', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-09 21:25:11.196105', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJwJ+E73YPLZm1OxWewpM2ZDiu1pmnGL7rta489O02oIAeHn7wmSYJk3TjQfoEv0GAtghOyLMLo5s92pNNwiaJo='} 2024-02-09T22:41:02.772 INFO:teuthology.orchestra.console.smithi006:Power off 2024-02-09T22:41:02.773 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-09T22:41:02.797 DEBUG:teuthology.orchestra.console.smithi006:power off output: Chassis Power Control: Down/Off 2024-02-09T22:41:02.798 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T22:41:02.819 INFO:teuthology.orchestra.console.smithi090:Power off 2024-02-09T22:41:02.820 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-09T22:41:02.830 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2024-02-09T22:41:02.843 DEBUG:teuthology.orchestra.console.smithi090:power off output: Chassis Power Control: Down/Off 2024-02-09T22:41:02.843 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T22:41:02.865 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2024-02-09T22:41:06.831 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T22:41:06.866 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T22:41:06.945 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2024-02-09T22:41:06.977 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2024-02-09T22:41:10.949 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T22:41:10.978 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T22:41:11.060 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2024-02-09T22:41:11.090 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2024-02-09T22:41:15.064 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T22:41:15.091 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T22:41:15.176 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is off 2024-02-09T22:41:15.204 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is off 2024-02-09T22:41:15.277 INFO:teuthology.orchestra.console.smithi006:Power off completed 2024-02-09T22:41:15.306 INFO:teuthology.orchestra.console.smithi090:Power off completed 2024-02-09T22:41:15.412 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 3397.9329159259796 failure_reason: 'Command failed on smithi006 with status 1: ''sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee26d820-c794-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c \''ceph versions | jq -e \''"\''"\''.mgr | keys\''"\''"\'' | grep $sha1\''''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=854a4cf6212d44adb2dea634b979eb15 status: fail success: false 2024-02-09T22:41:15.413 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-09T22:41:15.505 INFO:teuthology.run:FAIL